Verified answer

Given:

y = ∫ tan²(4x) /[(x-5)tan 4x + tan²(4x)]dx and

h = ∫ (4x² - 20x + 25)/[4x² + (2 tan4x - 20)x + 5 ( -tan 4x + 5)] dx

First, y can be reduced to y = ∫ tan(4x) /[(x-5)+ tan(4x)] dx.

Now let's rearrange h. Numerator can be written as complete square 4x² - 20x + 25 = (2x - 5)².

Let's factor the denominator:

4x² + (2 tan4x - 20)x + 5 ( -tan 4x + 5) = 0

D = (2 tan4x - 20)² - 4·4·5 ( -tan 4x + 5) = 4tan²4x - 80tan4x + 400 - 400 + 80tan4x = 4tan²4x

x1 = (-2tan4x + 20 + 2tan4x)/8 = 20/8 = 5/2 ,

x2 = (-2tan4x + 20 - 2tan4x)/8 = (20 - 4tan4x)/8 = (5 - tan4x)/2 ,

Then

4x² + (2 tan4x - 20)x + 5 ( -tan 4x + 5) = 4(x - 5/2)(x - 5/2 + (tan4x)/2) or

4x² + (2 tan4x - 20)x + 5 ( -tan 4x + 5) = (2x - 5)(2x - 5 + tan4x) .

Coming back to the integral h:

h = ∫ (2x - 5)²/[(2x - 5)(2x - 5 + tan4x)] dx = ∫ (2x - 5)/(2x - 5 + tan4x) dx .

So

y + h = ∫ tan(4x) /[(x-5)+ tan(4x)] dx + ∫ (2x - 5)/(2x - 5 + tan4x) dx.

I assume there is a typo in the denominator of the first integral, it should be [(2x-5)+ tan(4x)].

In such a case

y + h = ∫ (tan4x + 2x - 5)/(2x - 5 + tan4x) dx = ∫ dx = x + C.

Initial integral becomes ∫ (y + h) tan 4x dx = ∫ (x + C) tan 4x dx.