Htydjtk

I'm not able to find any similar triangles here, so how can I relate the areas?

Notation: $[..]$ denotes the area of the polygon "$...$"

Let $require{color}colorbox{yellow}{$[AEG]$}=Simplies require{color}colorbox{lightgreen}{$[DGF]$}=5S$. Let furthermore $[ABCD]=4T$. Thus
$$require{color}colorbox{pink}{$[AGD]$}=[AED]-require{color}colorbox{yellow}{$[AEG]$}=frac{[ABCD]}{4}-require{color}colorbox{yellow}{$[AEG]$}=T-S$$
Notice that $$[AFD]=frac{DA·DC}{2}=frac{[ABCD]}{2}$$ $$therefore [AFD]=require{color}colorbox{pink}{$[AGD]$}+require{color}colorbox{lightgreen}{$[DGF]$}=frac{[ABCD]}{2}=2Tiff require{color}colorbox{pink}{$[AGD]$}+require{color}colorbox{lightgreen}{$[DGF]$}=2T$$ $$iff T+4S=2Tiff 4S=Timplies require{color}colorbox{pink}{$[AGD]$}=3S$$

Observe now that $$frac{require{color}colorbox{pink}{$[AGD]$}}{require{color}colorbox{lightgreen}{$[DGF]$}}=frac{AG}{GF}=frac{require{color}colorbox{yellow}{$[AEG]$}}{[GEF]}iff frac{3S}{5S}=frac{S}{[GEF]}iff [GEF]=frac{5S}{3}$$

Since $E$ is the midpoint of $AB$, it's easy to prove that $[AEF]=[EBF]$. Hence $$[EBF]=require{color}colorbox{yellow}{$[AEG]$}+[GEF]=frac{8S}{3}implies require{color}colorbox{lightblue}{$[GEBF]$}=frac{13S}{3}$$ Finally

$$frac{require{color}colorbox{pink}{$[AGD]$}}{colorbox{lightblue}{$[GEBF]$}}=frac{3S}{frac{13S}{3}}=frac{9}{13}$$

The following figure shows acceptable coordinates:

The only relevant parameter is $cin>]0,1[>$. Compute the coordinates of the point $P$ in terms of $c$. It is then easy to find the areas of all involved areas in terms of $c$. This will lead to an equation for $c$ and then to a value for the ratio between the magenta and the blue area.

Note: If a triangle $triangle$ is spanned by the vectors $a=(a_1,a_2)$ and $b=(b_1,b_2)$ then its area is given by
$${rm area}(triangle)={1over2}|awedge b|={1over2}|a_1b_2-a_2b_1| .$$

Another way to find the desired ratio of magenta and blue sector is to determine the length of altitude of green and yellow triangle.

Let denote some of the segments of the rectangle such as $AB = x, AD = z, CF = y$

So, we get $BF = z-y$ and $AE = frac{x}{2}$. You were looking for some similar triangles and also put emphasis on it. So, let proceed with the similarity.

Notice that $triangle DAE sim triangle HGE$ and $triangle FBA sim triangle HGA$ and from their similarity, you can easily find $|HG|$ and also $|AG|$ and $|GE|$.

After that, $triangle DLN sim triangle HKN$ and similarly you get $|KH|$ and likewise $triangle DLN sim triangle DCF$. $|DF|$ can be determined through pythagorian theorem by applying in the right angled $triangle DCF$. Thus, we can find the area of both yellow and green triangle.

Next, after taking the proportion of area of $triangle DHA$ and $(triangle FHB + triangle BHE)$, evaluate the term whatever you got and the variable which I took such as $x, y $ and $z$ are surely to be nullified in this case. But, a problem is that I couldn't reach to the conclusion.

But I think, this method wouldn't be so much effective to you. Because, relating the area is the best process. The following method would be sometimes so much disturbing and complexed. So, I should highly suggest you to follow the solution provided by Dr. Mathva. It is quite elegant and largely acceptable.

HINT.-Let the rectangle be of sides $2a$ and $b$. Knowing the coordinates of all the involved points the solution is easy to find. Line $DE$ being fixed the problem is the determination of the point $F$ in the side $BC$. Put $A=(0,0),E=(a,0),B=(2a,0),F=(2a,x),D=(0,b)$.

The point $G$ is intersection of the lines $DE$ and $AF$ so $G=left(dfrac{2ab}{x+2b}, dfrac{bx}{x+2b}right)$.

Now use the formula for area of a triangle $triangle ABC=frac12detbegin{pmatrix} a_1&a_2&1\b_1&b_2&1\c_1&c_2&1end{pmatrix}$ from which, because of $dfrac{triangle DGF}{triangle AGE}=5,$ you have as a result $x=dfrac{4b}{7}$ then $G=left(dfrac{7a}{9},dfrac{2b}{9}right)$.

What remains comes from a direct calculation now (exactly as the first one adding in the denominator the area $dfrac{ax}{2}$ of the triangle $triangle EBF$. Note that the corresponding value is independent of the sides of the rectangle.