I don' t know what information you have about it, so i'll tell all i know about it : not very much actually and i don't fully understand it, so don't be surprised if it seems unclear or if you already know this...
The conversion is using the following equations:
On the other hand the setConvert function set various coefficients named k0 to k5 (s1.7 format). They are supposed to define a yuv/rgb conversion matrix but i don't know if it's the same thing as the Cx coefficients coz i can't see why it defines 6 constants while the equations only needs 5.
I know these equations. I need information about how to read Y,U,V values from rdram memory (How is a YUV texture stored in memory in bits). N64 appearently stores Y,U,V in 4:2:2 or 4:1:1 formats, or other similiar binary formats. I have tried many combinations, none works.