The 81-year-old stood down in January while the French government carried out an audit into the federation, BBC Sport reported.
"Considering his conduct towards women, his public comments and the governance failings of the FFF, Mr Le Graet no longer has the necessary legitimacy to run and represent French football," the report concluded when it was published earlier this month.
It also said Le Graet, who has always denied any wrongdoing, should not return to his role because his "behavioural excesses are incompatible with the carrying out of his functions".
Several members of the FFF have said that Le Graet will continue his role with world governing body FIFA.
In January 2022, he was appointed as FIFA president Gianni Infantino's delegate in the organisation's new office in Paris.
Lyon president Jean-Michel Aulas said Infantino had "seized the opportunity to use Noel's capabilities", adding "if that means indirectly that it comes at a good time for Noel, it's a good thing".
Le Graet, who was in office for 11 years and had a mandate until 2024, has previously faced claims of sexual harassment, which he denies.
There were also calls for him to quit over his comments about Zinedine Zidane, who won the World Cup and Euros as a player for France and guided Real Madrid to three successive Champions League titles.
Le Graet said he would not take a call from Zidane about becoming France coach before apologising for his "clumsy" remarks.
The French government began the audit in October after being made aware of sexual harassment and bullying allegations at the organisation, which the FFF has denied.
Announcing his resignation on Tuesday, the FFF said it "recognises the remarkable sporting and economic performance of Noel le Graet".
Referring to the audit, it added: "The FFF nevertheless notes that the report is based less on objective facts than on comments that have sometimes led to exaggerated bad-mouthing of the body."
In an interview published later on Tuesday, Le Graet told French sports daily L'Equipe his lawyers would challenge the sports ministry's report, and insisted he had "never harassed anyone".
Le Graet took office in 2011, with the men's national team winning the 2018 World Cup and finishing as runners-up at Euro 2016 and the 2022 World Cup.
Before he stood down earlier this year, Le Graet extended Didier Deschamps' contract as manager of the national side until 2026.
Vice-president Philippe Diallo will remain as interim president until June when a permanent successor will be chosen.
Le Graet's resignation comes after captain of the women's team Wendie Renard and two other players said they will not play at this summer's World Cup and criticised the country's football "system".
France are due to play at the Women's World Cup later this summer and Diallo said a decision on head coach Corinne Diacre's future would be taken "very soon".