Menu
What's new
By:
Filters

Hey You! Pikachu - Possible HLE Implementation

OP
F

Falkoner

New member
I would love it if someone were willing to take that up, I do admit though that I'm really just a web developer who learned to code initially in C/C++, so reverse-engineering is outside my expertise :( I suppose that's why I'm going the brute-forcing route, since I'm not sure how to even begin actually understanding the VRU's functionality.

I tried looking at the SDK, but I could only track down the function calls in the Manual, but I would love it if someone more skilled wants to check it out. I honestly think the VRU response codes should be in the ROM if someone were to reverse-engineer it, and that would essentially take the VRU problem out of the equation and turn it into a software voice recognition issue, which I could potentially solve.
 
Last edited:
OP
F

Falkoner

New member
Hey You, Pikachu! Mic Test

So I went ahead and built a dictionary for Hey You! Pikachu using PocketSphinx, it works pretty well (about as well as the original game did). Needs some tweaking for commonly used words and Pokemon names (Pikachu is hit and miss), but otherwise it's actually quite good.

Run the batch file and say any of the in-game commands into your mic to see if it properly detects it.

I just need to look into tweaking the dictionary to be more precise for certain words and stop it from attempting to find more than one phrase used at once. Of course, still needing someone to find the VRU output for each phrase to pair it with this setup.
 
Last edited:
N

NotCow

New member
Falkoner said:
Hey You, Pikachu! Mic Test
Click to expand...

Wow, that's incredible! It's really good at voice recognition, but it also picks up environment noise as unrelated words, and it rarely registers "Pikachu" correctly...but whatever, I've waited years for a HYP! emulator, it's really crazy to see something like that and it's close enough.

Thanks for your work man, I don't know how soon you plan to continue work on this but it's really cool to see that kind of progress; for someone to potentially make that game playable for people without the VRU.
 
OP
F

Falkoner

New member
NotCow said:
Wow, that's incredible! It's really good at voice recognition, but it also picks up environment noise as unrelated words, and it rarely registers "Pikachu" correctly...but whatever, I've waited years for a HYP! emulator, it's really crazy to see something like that and it's close enough.

Thanks for your work man, I don't know how soon you plan to continue work on this but it's really cool to see that kind of progress; for someone to potentially make that game playable for people without the VRU.
Click to expand...

Yeah, my hope is that by having this side of things essentially completed that people who are more skilled than me at reverse-engineering, or someone who owns an Adaptoid will take up the other side of the issue, communicating with the game itself.

I figure the original game likely also had major issues with background noise, but since you only turn it on right when you need it that should make it less of a problem, the issue is that it's currently trying to constantly detect phrases, and only does the output when it hears silence, that'll of course be different in the final implementation.
 
Last edited:
T

theCreeper8020

New member
u really need to link the hty with the game/ edit the roms code so u can type insted

Falkoner said:
Yeah, my hope is that by having this side of things essentially completed that people who are more skilled than me at reverse-engineering, or someone who owns an Adaptoid will take up the other side of the issue, communicating with the game itself.

I figure the original game likely also had major issues with background noise, but since you only turn it on right when you need it that should make it less of a problem, the issue is that it's currently trying to constantly detect phrases, and only does the output when it hears silence, that'll of course be different in the final implementation.
Click to expand...
it says it all in the tittle of this comment

- - - Updated - - -

like making a new rom.
 
C

cymon

New member
Falkoner said:
Hey You, Pikachu! Mic Test

So I went ahead and built a dictionary for Hey You! Pikachu using PocketSphinx, it works pretty well (about as well as the original game did). Needs some tweaking for commonly used words and Pokemon names (Pikachu is hit and miss), but otherwise it's actually quite good.

Run the batch file and say any of the in-game commands into your mic to see if it properly detects it.

I just need to look into tweaking the dictionary to be more precise for certain words and stop it from attempting to find more than one phrase used at once. Of course, still needing someone to find the VRU output for each phrase to pair it with this setup.
Click to expand...

Any chance for a total idiots guide to setting this up for Project64? If not P64 is there a better emulator that I should be using?
 
V

V1del

New member
This isn't implemented yet for use in any emulator, this is just the recognition groundwork that has nothing to do with emulation. Implementing support in an emulator is something that still has to be done by someone (and is likely to be an humongous task, so don't hold your breath)
 
U

ulao

Member
Whats up guys. Ulao from Bliss-Box here... So I'm going to add the VRU to my Bliss-Box. I found the main commands to make the units start working. I'll be working on this to try and get it to recognize words and see what codes it wants to send. The best I can tell so far is that it sends 11 bytes. I'm guessing that is ASCII data or just some hex codes.
example (11 bytes) all zeroes.
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
If each byte was a word then you could send lots of data, if each byte is an ASCII char it would take a lot more. If its just 11 bytes I can send all of that down the USB interface as I have a total of 13 packets exposed to HID DX.

At this point I just need to know what goes on in the rom. I dug up some old MESS emulator dev talking about it. He saw this at boot up.
The 0x0a command sends 16 bytes to the controller and expects 1 byte in return.
The 0x0b command sends 2 bytes to the controller and expects 3 bytes in return.
The 0x0c command sends 6 bytes to the controller and expects 1 byte in return.
The 0x0d command sends 2 bytes to the controller and expects 1 byte in return.

I see most of that on my end as well but 0x0b returns 11 bytes for me.

I think we need to know more about what is going on in game. I'm hopping the game is a listen only device, so no game instruction needs to talk to the adapter. If it does need to send data to the adapter we could always use the rumble data. Just send either custom rumble or unused effect types. If anyone wants to help just jump in. If there are any Bliss-Box owners out there and want to beta test let me know.
 
OP
F

Falkoner

New member
Based on the description of the device here, it appears that the only major instructions are an instruction from the CPU to the game to begin recording(when player holds down Z), the instruction to retrieve the captured data and word guess from the VRU, when Z is released, and the initialization instructions sent to the VRU, which appear to essentially be the word dictionary, which the VRU uses to guess which word the player said. The captured data itself doesn't appear to ever be sent back to the N64, just what word the VRU thought it was.

Issues opened up in both Mupen64Plus and Project64 repositories, since I believe we need modifications to controller plugins on both to get it working. Project64's implementation of NRage's plugin likely is the closest, since it has some kind of RAW input support implemented for Adaptoids, and has gotten controller paks working as well but I'm uncertain that is sufficient for the VRU to work.
Mupen64+ Issue
Project64 Issue

I'm fairly certain it needs to be done, because I purchased a Wishtec Adaptoid, installed a WinXP environment to get the driver working, and found that there was still no support in the emulators for the VRU.

I PM'd MdkCheatz back in December, and he responded, here's what he said he knew:
The VRU was tech designed originally to be compatible for use with future games.

It's been a while since I've been involved in the Hey You Pikachu project but I reckon you must have come across the research I had once published back when I was trying to reverse engineer it.

I see someone is trying to do achieve what I was once hoping to do.

Here's the deal... Anything I had done was surface fluff. All that I know is theoretical based on interpreting developer data I found online regarding the VRU.

Regarding the VRU... you have 4 very basic concepts.

1. When instructed by the game (any game), the VRU hardware begins writing and translating analogue data taken from the mic itself.

2. The VRU goes through a matching process inside the VRU and not the game. This word or phrase is translated into instructional code back to the game.

3. The word phrase is calculated as a probability of match. I forget if this happens in the VRU or the game itself.

4. The game has a "tolerance" range of probability it is willing to accept depending on a difficulty setting.


Basically, the VRU turned out to be a psuedo voice recognition unit. It isn't true recognition. thats why I stopped doing research on it. To emulate the VRU would involve learning how to make a true voice recognition unit and trying to water it down to a simple set of instructions. IMO a complete waste of time. But if you are ambitious enough to do this, hats off to you.

A feasible means to emulate the VRU would likely be forcing a probability range that is accepted by the game. you could theoretically play the game to the end but you would loose the value of actually playing it. It would be as if someone is playing it for you.

Or, you could offer a solution specific to the game where the player can select input based on random options of words (like multiple choice),therefore adding the possibility of more involvement with the player. but the real magic would still be lost.
Click to expand...

Doing a little more digging, here's the official N64 Programming Manual regarding the VRU. I think that's actually perhaps the most useful piece of info I've found yet. Not certain if everything is accurate, since this appears to apply to the Japanese VRU, but I think it should be a very similar process.

If I'm interpreting it correctly, each word registered is an 8-byte character, here's the relevant code for the entire process, from initialization to creating the dictionary, to adding words, to then starting the recording, to receiving the result back.
s32 osVoiceInit(OSMesgQueue *siMessageQ, OSVoiceHandle *hd, int channel);
typedef struct {
OSMesgQueue *__mq; /* SI message queue */
int __channel; /* Controller port No. */
s32 __mode; /* Used within the OS */
u8 cmd_status; /* Command status */
} OSVoiceHandle;
s32 osVoiceSetWord(OSVoiceHandle *hd, u8 *word);
s32 osVoiceClearDictionary(OSVoiceHandle *hd, us words);
s32 osVoiceSetWord(OSVoiceHandle *hd, u8 *word);
s32 osVoiceStartReadData(OSVoiceHandle *hd);
s32 osVoiceGetReadData(OSVoiceHandle *hd, OSVoiceData *result);
typedef struct {
u16 warning; /* Warning */
u16 answer_num; /* Candidate number (0~5) */
u16 voice_level; /* Voice input level */
u16 voice_sn; /* Relative voice level */
u16 voice_time; /* Voice input time */
u16 answer[5]; /* Candidate word number */
u16 distance[5]; /* Distance value */
} OSVoiceData;
s32 osVoiceStopReadData(OSVoiceHandle *hd);
//Registered Words:
u8 *registration_word[] = {
"yakiniku",
"mario",
.
.
.
"pikachu"
};
Click to expand...

What interested me is this says that 255 syllables can be registered, including multiple syllables per word, but 640 commands appear to be in the game's code, with 459 of them being unique(I assume the repeats are pronunciation options). I think they must reregister the dictionary depending on what section of the game you're in, pinata commands in the pinata game, vegetable names in the soup-making level, Pokemon names are only registered as needed(ie, for Who's That Pokemon! game), etc. That potentially complicates the HLE implementation a bit, but if we can get the Sphinx dictionary accurate enough I figure it should work fine.

f26-08-05.gif
 
Last edited:
U

ulao

Member
Based on this

1. When instructed by the game (any game), the VRU hardware begins writing and translating analogue data taken from the mic itself.

2. The VRU goes through a matching process inside the VRU and not the game. This word or phrase is translated into instructional code back to the game.

3. The word phrase is calculated as a probability of match. I forget if this happens in the VRU or the game itself.

4. The game has a "tolerance" range of probability it is willing to accept depending on a difficulty setting.
Click to expand...

and other tid-bits from what you said it looks like my suspicions were correct ( and that's a good thing). I could emulate all of that without the game. I could constantly send the recognized word to the game via game-pad buttons. No plug-in would be needed, just core code. Once the games gets a z-button press it will then look for what is being sent. I could have the adapter send the last command up to a given time to ensure it was received. Obviously this could be better refined in the way I mentioned above.

game sees a z-button
pj64 sends a custom rumble command to indicate it's expecting data
bliss-box see this command and has the vru capture for x seconds.
bliss-box sends the recognized code back via button data.

Personally I really do not see all of that necessary except for the different modes we may need to put this vru in. Either way I think I can fill in the blanks from the data you sent above and make this thing work. Once I get it to receive commands ill report back. Then it is just a matter of how to listen for it and when.

Yes the Japaneses part of it is unfortunate because it is a jap-char map not letters I know. Though I do also believe I may not need that. The doc does not go low level enough and show the physical layer but maybe I can guess some of it. I think finding the patent on this may help (I have yet to search).

Thx Falkoner, great info!
 
Last edited:
OP
F

Falkoner

New member
So you think the custom rumble feature in NRage is sufficient to send the dictionary to the VRU during initialization? My only other concern is that the game appears to recieve live feedback from the VRU while you are speaking, an undocumented feature as far as I've seen.
 
U

ulao

Member
So you think the custom rumble feature in NRage is sufficient to send the dictionary to the VRU during initialization?
Click to expand...
Oh absolutely. The Bliss-Box uses custom rumble to send API commands. Like memory pack transfers and such. The unfortunate thing is that no devs have tried to use it. For example I use this very same thing for DC emulation, like sending LCD data. You can see pictures of that on the Bliss-Box pages. The possibilities are simply endless and the best part is no friggen drivers.



My only other concern is that the game appears to recieve live feedback from the VRU while you are speaking, an undocumented feature as far as I've seen.
Click to expand...
This is not Unachievable. The latency from VRU to game (emulator) is 16 ms. The Eye can detect near 17ms latency but the ear is much much greater. From game to VRU (via FFB custom) and depending on data will be significantly more (maybe 50ms). but again this is not really that bad when it comes to speech. IMO the data the game needs to send to the VRU is going to be a byte or maybe two bytes and in that case the latency will be 8ms. So round trip in my estimation (allowing for hiccups) is about 32ms tops and 16ms on average. As for sending the dictionary, latency is a non issue so it can take its time, but the Bliss-Box API will allow unlimited data transfers.
 
OP
F

Falkoner

New member
ulao said:
Oh absolutely. The Bliss-Box uses custom rumble to send API commands. Like memory pack transfers and such. The unfortunate thing is that no devs have tried to use it. For example I use this very same thing for DC emulation, like sending LCD data. You can see pictures of that on the Bliss-Box pages. The possibilities are simply endless and the best part is no friggen drivers.
Click to expand...

Absolutely beautiful, that'll also make this solution quite effective at being cross-platform if Mupen64Plus can get us a basic API similar to the one in NRage.


ulao said:
This is not Unachievable. The latency from VRU to game (emulator) is 16 ms. The Eye can detect near 17ms latency but the ear is much much greater. From game to VRU (via FFB custom) and depending on data will be significantly more (maybe 50ms). but again this is not really that bad when it comes to speech. IMO the data the game needs to send to the VRU is going to be a byte or maybe two bytes and in that case the latency will be 8ms. So round trip in my estimation (allowing for hiccups) is about 32ms tops and 16ms on average. As for sending the dictionary, latency is a non issue so it can take its time, but the Bliss-Box API will allow unlimited data transfers.
Click to expand...

That also seems about on-par with the delays they have already inherent in the design:
f26-08-03.gif


I look forward to hearing when you've gotten it receiving dictionary words, that'll be a major breakthrough!
 
U

ulao

Member
Lots of this info you have is of great help but sadly none of this info tells what the commands are. For example CONT_ERR_NO_CONTROLLER must be a define of some kind of command.
example:
CONT_ERR_NO_CONTROLLER 0x01
of course that is just a guess. If this info can be had anywhere it is key to this development. The easiest and best way is to get with a emulator developer. As soon as the Z-button is pressed that game rom will send data on hardware level of some kind. Much like I quoted above on MESS.

Anyways just throwing that out there. I have the hardware all set up and the Bliss-box is detecting the vru and initializing what I think is USA language. I was also wondering does the vru require that you plug it in port 4?
 
Last edited:
OP
F

Falkoner

New member
Well, it appears that the port # is configurable, from the OSVoiceHandle struct, but in the case of Hey You, Pikachu! it required Port 4, with a controller plugged into Port 1.

I believe most of those errors shouldn't appear in Hey You, Pikachu, since they were mostly intended for testing, not errors that should crop up once the game is working properly(except the no controller error you pointed out.)

I'm searching for some more detailed information, and I'll try to get that to you shortly.

I think here is the struct for those error messages:
Code: 
typedef struct {
    u16type;  /*Controller type*/
    u8status; /*Controller Pak status*/
    u8errno;  /*Error from SI device
}OSContStatus;

And meanings:
Voice Recognition System Errors

No SI device inserted

CONT_ERR_NO_CONTROLLER
Nothing is connected to the controller port.
Different SI Device is inserted

CONT_ERR_DEVICE
Something other than the Voice Recognition System is connected to the controller port.
Data transfer failure

CONT_ERR_CONTRFAIL
There was a data transmission failure. There is a problem in the Voice Recognition System connection.
Errors specific to this particular SI device
CONT_ERR_VOICE_WORD
A word containing improper characters has been registered. The set word is invalidated and the word number is not incremented.
CONT_ERR_VOICE_MEMORY
Dictionary memory overflow. However, if the recognition command is executed in this condition, normal recognition processing can be performed even if the number of words which have been set is less than the number of words set by the osVoiceClearDictionary() function.
CONT_ERR_NOT_READY
Either no voice has been input, or results cannot be acquired for some reason, such as that processing is still underway, etc.
Fatal error

CONT_ERR_INVALID
There is an error in the function call method or in the argument.
CONT_ERR_VOICE_NO_RESPONSE
There was no response from the Voice Recognition System. There may be a problem with the hardware.
Click to expand...

The voice demo from the SDK might have more information, I'm trying to sort through and find useful info, here's some code:
  Spoiler:
Code: 
/*---------------------------------------------------------------------
 	Copyright (C) 1998 Nintendo.
 	
 	File		main.c
 	Coded    by	Tetsuyuki Ootsuka.	July, 1998.
 	Comments	Voice Recognition System Demo Program
   
 	$Id: main.c,v 1.4 1999/04/16 10:14:40 yoshida Exp $
   ---------------------------------------------------------------------*/
/**************************************************************************
 *
 *  $Revision: 1.4 $
 *  $Date: 1999/04/16 10:14:40 $
 *  $Source: 
 *
 **************************************************************************/
#include <ultra64.h>
#include "nu64sys.h"
#include "graph.h"
#include "action.h"
#include "siproc.h"

#include "mes.h"

#define TITLE_MES_X     40
#define TITLE_MES_Y     20
#define STATUS_X        40
#define STATUS_Y        60
#define WORD_X          48
#define WORD_Y          64
#define READY_X         40
#define READY_Y         180
#define ERR_X           8
#define ERR_Y           100
#define WARN_X          40
#define WARN_Y          200

OSVoiceData	result;
OSVoiceHandle	vhd;

u16		*gp;
u8		draw_buffer = 1;

static OSThread siThread;
static u64      siStack[STACKSIZE/sizeof(u64)];
static OSMesg   sendMsgBuf[SI_MSG_NUM];
OSMesgQueue     sendMsgQ;

     
/*---------------------------------------------------------------------
                  main
  ---------------------------------------------------------------------*/

void
mainproc(void) {

  u32		i;
  u8    	mode, si_mode;
  s32   	ret;
  u16		color;
  u16		delta_x;
  u8		tmp[16];
  sendMsg 	sendMesgBuff;
  sendMsg	*sendMesg = &sendMesgBuff;
  retMsg 	dummyBuff;
  retMsg 	*dummy = &dummyBuff;


  /*-- Initializing SI thread --*/

  osCreateMesgQueue(&sendMsgQ, sendMsgBuf, SI_MSG_NUM);
  osCreateThread(&siThread, 8, (void *)siproc, 0,
		 siStack+STACKSIZE/sizeof(u64), 12);
  osStartThread(&siThread);

  ret = 0;
  mode = InitMode;
  si_mode = VOICE_END;

  delta_x = 0;
  result.answer_num = 0;
  vhd.cmd_status = 0xff;


  /*-- Main Loop --*/

  while(1){


    /*-- Writing back frame buffer --*/
    
    osWritebackDCache(cfb[draw_buffer], 2*SCREEN_WD*SCREEN_HT);

    /*-- Registering pointer on frame buffer --*/

    osViSwapBuffer(cfb[draw_buffer]);
    
    /*-- Waiting for vertical re-trace --*/

    while(osRecvMesg(&retraceMessageQ, NULL, OS_MESG_NOBLOCK) == 0);
    osRecvMesg(&retraceMessageQ, NULL, OS_MESG_BLOCK);

    /*-- Setting frame buffer pointer --*/

    draw_buffer ^= 1;
    gp = (u16 *)cfb[draw_buffer];
    gcls(gp);



    /***** Obtaining controller data (executing SI thread) *****/

    if((si_mode == VOICE_END) || (mode == NoContMode && si_mode == CONTR_END)){
      si_mode = CONTR_START;
      sendMesg->mode = mode;
      sendMesg->si_mode = si_mode;
      osSendMesg(&sendMsgQ, (OSMesg)sendMesg, OS_MESG_NOBLOCK);
    }
    
    /*-- Obtaining controller status --*/

    if(si_mode == CONTR_START){
      if(osRecvMesg(&retMsgQ, (OSMesg *)&dummy, OS_MESG_NOBLOCK) == 0){
	si_mode = dummy->si_mode;
      }
    }
    
    if(si_mode == CONTR_END){
      
      for(i = 0;i < MAXCONTROLLERS;i++){

	/*-- Retrying from initialization --*/

	if(Ac.pad[i].push & A_BUTTON){
	  mode = InitMode;
	  result.answer_num = 0;
	  break;
	}

	/*-- Halt the recognition process --*/

	if((Ac.pad[i].push & B_BUTTON) && (mode == GetReadMode)){
	  mode = StopReadMode;
	  break;
	}
      
      }
    
    }
    

    /***** Starting voice recognition process (executing SI thread) *****/

    if((si_mode == CONTR_END) && (mode != NoContMode)){
      si_mode = VOICE_START;
      sendMesg->mode = mode;
      sendMesg->si_mode = si_mode;
      osSendMesg(&sendMsgQ, (OSMesg)sendMesg, OS_MESG_NOBLOCK);
    }
    
    /*-- Completing voice recognition process --*/

    if(si_mode == VOICE_START){
      if(osRecvMesg(&retMsgQ, (OSMesg *)&dummy, OS_MESG_NOBLOCK) == 0){
	ret = dummy->ret;
	mode = dummy->mode;
	si_mode = dummy->si_mode;
      }
    }
    

    /***** Display recognized result *****/

    printkanji(gp, TITLE_MES_X, TITLE_MES_Y, WHITE, mes0);


    /*-- Display process halt status --*/

    if(mode == NoContMode){
      printkanji(gp, WARN_X, WARN_Y, YELLOW, mes1);
      prints(WARN_X+16*4, WARN_Y+16, RED, "-- Push A Button ! --");
    }

    
    /*-- Display error result --*/

    if(ret == CONT_ERR_NO_CONTROLLER){
      printkanji(gp, ERR_X+80, ERR_Y, RED, mes2);
      printkanji(gp, ERR_X+80, ERR_Y+16, RED, mes3);
      prints(ERR_X+56, ERR_Y+48, RED, "(CONT_ERR_NO_CONTROLLER)");
      mode = NoContMode;
      continue;
    }
    else if(ret == CONT_ERR_DEVICE){
      printkanji(gp, ERR_X+40, ERR_Y, RED, mes4);
      printkanji(gp, ERR_X+88, ERR_Y+16, RED, mes5);
      prints(ERR_X+84, ERR_Y+48, RED, "(CONT_ERR_DEVICE)");
      mode = NoContMode;
      continue;
    }
    else if(ret == CONT_ERR_VOICE_NO_RESPONSE){
      printkanji(gp, ERR_X+64, ERR_Y, RED, mes6);
      printkanji(gp, ERR_X+88, ERR_Y+16, RED, mes7);
      prints(ERR_X+40, ERR_Y+48, RED, "(CONT_ERR_VOICE_NO_RESPONSE)");
      mode = NoContMode;
      continue;
    }
    else if(ret == CONT_ERR_CONTRFAIL){
      printkanji(gp, ERR_X+72, ERR_Y, RED, mes8);
      prints(ERR_X+72, ERR_Y+32, RED, "(CONT_ERR_CONTRFAIL)");
      mode = NoContMode;
      continue;
    }
    else if(ret == CONT_ERR_INVALID){
      printkanji(gp, ERR_X+80, ERR_Y, RED, mes9);
      prints(ERR_X+80, ERR_Y+32, RED, "(CONT_ERR_INVALID)");
      mode = NoContMode;
      continue;
    }
    else if(ret == CONT_ERR_VOICE_WORD){
      printkanji(gp, ERR_X+80, ERR_Y, RED, mes10);
      printkanji(gp, ERR_X+72, ERR_Y+16, RED, mes11);
      prints(ERR_X+68, ERR_Y+48, RED, "(CONT_ERR_VOICE_WORD)");
      mode = NoContMode;
      continue;
    }
    else if(ret == CONT_ERR_VOICE_MEMORY){
      printkanji(gp, ERR_X+48, ERR_Y, RED, mes12);
      prints(ERR_X+60, ERR_Y+32, RED, "(CONT_ERR_VOICE_MEMORY)");
      mode = NoContMode;
    }
    else if(ret == CONT_ERR_NOT_READY){

      if((vhd.cmd_status == VOICE_STATUS_START) || (vhd.cmd_status == VOICE_STATUS_CANCEL)){
	printkanji(gp, READY_X, READY_Y, CYAN, mes13);
	mode = GetReadMode;
      }
      
    }

    
    /*-- Display recognized result --*/

    if(result.answer_num != 0){

      /*-- Warning --*/

      if(result.warning != 0){

	delta_x = 0;
	
	if(result.warning & VOICE_WARN_TOO_NOISY){
	  printkanji(gp, WARN_X, WARN_Y, YELLOW, mes14);
	  delta_x += 64;
	}
	if(result.warning & VOICE_WARN_NOT_FIT){
	  printkanji(gp, WARN_X+delta_x, WARN_Y, YELLOW, mes15);
          delta_x += 64;
	}
	if(result.warning & VOICE_WARN_TOO_LARGE){
	  printkanji(gp, WARN_X+delta_x, WARN_Y, YELLOW, mes16);
          delta_x += 64;
	}
	if(result.warning & VOICE_WARN_TOO_SMALL){
	  printkanji(gp, WARN_X+delta_x, WARN_Y, YELLOW, mes17);
	}

      }
      
      /*-- Display recognized words --*/

      color = WHITE;
      
      for(i = 0;i < 5;i++){

	if(result.answer[i] == 0x7fff){
	  break;
	}
	
        if(i >= result.answer_num){
	  color = YELLOW;
	}
	
	printkanji(gp, WORD_X, (int)(WORD_Y+16*(i+1)), color, mes[result.answer[i]]);
	sprintf(tmp, "%4d", result.distance[i]);
	prints(WORD_X+16*12, (int)(WORD_Y+16*(i+1)), color, tmp);

      }

    }
    

    /*-- Display status --*/

    switch(vhd.cmd_status){
      
    case	VOICE_STATUS_READY:
      
      printkanji(gp, STATUS_X, STATUS_Y, WHITE, mes18);
      break;
      
    case        VOICE_STATUS_START:

      printkanji(gp, STATUS_X, STATUS_Y, WHITE, mes19);
      break;

    case        VOICE_STATUS_CANCEL:

      result.answer_num = 0;
      
      printkanji(gp, STATUS_X, STATUS_Y, WHITE, mes20);
      break;

    case        VOICE_STATUS_BUSY:

      printkanji(gp, STATUS_X, STATUS_Y, WHITE, mes21);
      break;

    case        VOICE_STATUS_END:

      printkanji(gp, STATUS_X, STATUS_Y, WHITE, mes22);
      break;
      
    }

  }
  
}

/*---------------------------------------------------------------------
        Copyright (C) 1998 Nintendo.
        
        File            siproc.c
        Coded    by     Tetsuyuki Ootsuka.      July, 1998.
        Comments        SI Device Procedure
   
        $Id: 
   ---------------------------------------------------------------------*/
/**************************************************************************
 *
 *  $Revision: 1.3 $
 *  $Date: 1998/10/02 09:02:38 $
 *  $Source: 
 *
 **************************************************************************/

#include <ultra64.h>
#include "nu64sys.h"
#include "siproc.h"
#include "action.h"

#include "message.c"

static OSMesg   retMsgBuf[RET_MSG_NUM];
OSMesgQueue     retMsgQ;

OSContStatus    contStatus[MAXCONTROLLERS];
OSContPad       contPad[MAXCONTROLLERS];
u8              contExist;
u8              contNo = 0;

u8              voice_size = sizeof(voice_data)/sizeof(u8 *);


/*---------------------------------------------------------------------
                  Serial Interface Procedure
  ---------------------------------------------------------------------*/
void
siproc(void)
{
  u32  		i;
  u8		mode, si_mode;
  s32		ret;
  retMsg 	retMesgBuff;
  retMsg 	*retMesg = &retMesgBuff;
  sendMsg 	dummyBuff;
  sendMsg 	*dummy = &dummyBuff;
  
  osCreateMesgQueue(&retMsgQ, retMsgBuf, RET_MSG_NUM);

  /*-- Initializing Controller  --*/

  osContInit(&siMessageQ, &contExist, contStatus);
  actionInit();


  while(1){


    osRecvMesg(&sendMsgQ, (OSMesg *)&dummy, OS_MESG_BLOCK);

    mode = dummy->mode;
    si_mode = dummy->si_mode;
    
    
    if(si_mode == CONTR_START){
      
      /***** Obtaining Controller Data *****/

      osContStartReadData(&siMessageQ);
      osRecvMesg(&siMessageQ, NULL, OS_MESG_BLOCK);
      osContGetReadData(contPad);
      actionUpdate();

      si_mode = CONTR_END;
      
    }
    else if(si_mode == VOICE_START){

      /***** Starting Voice Recognition Process  *****/
      
      switch(mode){
	
	/*-- Initialization --*/
	
      case	InitMode:

	osContStartQuery(&siMessageQ);
	osRecvMesg(&siMessageQ, NULL, OS_MESG_BLOCK);
	osContGetQuery(contStatus);
	
	for(i = 0; i < MAXCONTROLLERS; i++){
	  if((contStatus[i].errno == 0) && ((contStatus[i].type & CONT_TYPE_MASK) == CONT_TYPE_VOICE)){
	    contNo = i;
	    break;
	  }
	}
	
	ret = osVoiceInit(&siMessageQ, &vhd, contNo);
	
#ifdef CHECKGAIN
	ret = osVoiceControlGain(&vhd, 1, 7);
#endif
	
	mode = ClrDicMode;
	break;
	
	/*-- Clearing Dictionary --*/
	
      case	ClrDicMode:
	
	ret = osVoiceClearDictionary(&vhd, voice_size);
	
	mode = SetWordMode;
	break;
	
	/*-- Setting Words for Dictionary --*/
	
      case	SetWordMode:
      
	i = 0;
    
	while(i < voice_size){

#ifdef CHECKWORD
	  if((ret = osVoiceCheckWord(voice_data[i])) != 0){
	    break;
	  }
#endif
		  
	  if((ret = osVoiceSetWord(&vhd, voice_data[i])) != 0){
	    break;
	  }
	   
	  i++;
	}
	 
	mode = StartReadMode;
	break;

	/*-- Start Voice Recognition --*/

      case	StartReadMode:
      
	ret = osVoiceStartReadData(&vhd);
	 
	mode = GetReadMode;
	break;

	/*-- Obtaining Recognition Result --*/

      case	GetReadMode:

	ret = osVoiceGetReadData(&vhd, &result);

	if(ret == 0){
	  mode = StartReadMode;
	}
      
	break;

	/*-- Halt the Process --*/

      case	StopReadMode:
      
	ret = osVoiceStopReadData(&vhd);

	mode = NoContMode;
	
        result.warning = 0;
	break;
      
      }

      si_mode = VOICE_END;

    }
    
    retMesg->ret = ret;
    retMesg->mode = mode;
    retMesg->si_mode = si_mode;
    
    osSendMesg(&retMsgQ, (OSMesg)retMesg, OS_MESG_BLOCK);
    
  }
  
}

/*---------------------------------------------------------------------*
        Copyright (C) 1998 Nintendo.
        
        $RCSfile: message.c,v $
        $Revision: 1.2 $
        $Date: 1998/09/25 21:51:51 $
 *---------------------------------------------------------------------*/u8	*voice_data[] = {
  "Yakiniku",   /*B.B.Q.*/
  "Nintendo",
  "Pikachu",
  "Genkidechu", /*I'm fine*/
  "Uchu",       /*Universe*/
  "MoshiMoshi",
  "Konnichiwa", /*Good afternoon*/
  "Konbanwa",   /*Good eveninig*/
  "Sayounara",  /*Good bye*/
  "Oyasumi",    /*Good night*/
  "ByeBye",
  "Mario",
  "Zelda",
  "Bento",      /*Lunch Box*/
  "Tendon",     /*Tempra rice bowl*/
  "Katudon",    /*Pork rice bowl*/
  "Ramen",      /*Noodle*/
  "Kyabetsu",    /*Cabbage*/
  "Tomato",
  "Daikon",     /*Radish*/
  "Ninjin",     /*Carrot*/
  "Peaman",     /*Green pepper*/
  "Tamanegi",   /*Onion*/
  "Jyagaimo",   /*Potato*/
  "Satsumaimo", /*Sweet potato*/
  "Renkon",     /*Lotus root*/
  "Cocoa",      /*Hot chocolate*/
  "Sutoro",     /*Straw*/
  "Ozon",       /*Ozone*/
  "Gobou",      /*Burdock*/
  "Udon",       /*Thick Noodle*/
  "Hoon",       /*Heat retaining*/
  "Budou",      /*Grape*/
  "Futon",
  "Ringo",      /*Apple*/
  "Melon",
  "Ichigo",     /*Strawberry*/
  "Fonto",      /*Font*/
  "Hontou",     /*Real*/
  "Daibouken",  /*Adventure*/
  "Fax",
  "Fox",
  "Ue",         /*Up*/
  "****a",      /*Down*/
  "Hidari",     /*Left*/
  "Migi",       /*Right*/
};
/*----------------------------------------------------------------
 [voice recognition testing program
----------------------------------------------------------------*/
short int mes0[]={0,128,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,-1};

/*----------------------------------------------------------------
 Halt
----------------------------------------------------------------*/
short int mes1[]={1792,1920,-1};

/*----------------------------------------------------------------
 Voice recognition system is not
----------------------------------------------------------------*/
short int mes2[]={128,256,384,512,2048,768,640,1536,2176,-1};

/*----------------------------------------------------------------
 connected
----------------------------------------------------------------*/
short int mes3[]={2304,2432,2560,2688,2816,2944,3072,3200,3328,-1};

/*----------------------------------------------------------------
Unknown device is
----------------------------------------------------------------*/
short int mes4[]={128,256,384,512,2048,768,640,1536,3456,3584,3712,3840,3712,2176,-1};

/*----------------------------------------------------------------
connected
----------------------------------------------------------------*/
short int mes5[]={2304,2432,2560,2688,2816,2944,3072,3968,-1};

/*----------------------------------------------------------------
 Voice recognition system 
----------------------------------------------------------------*/
short int mes6[]={128,256,384,512,2048,768,640,1536,4096,4224,3712,-1};

/*----------------------------------------------------------------
 is not responding
----------------------------------------------------------------*/
short int mes7[]={4352,4480,2176,4608,4736,3072,3200,3328,-1};

/*----------------------------------------------------------------
 Misconnection
----------------------------------------------------------------*/
short int mes8[]={2304,2432,4864,4992,5120,2176,4608,4736,3072,3968,-1};

/*----------------------------------------------------------------
 Fatal error
----------------------------------------------------------------*/
short int mes9[]={5248,5376,5504,5632,5760,1408,5888,6016,3968,-1};

/*----------------------------------------------------------------
 Registering invalid
----------------------------------------------------------------*/
short int mes10[]={6144,6272,4864,6400,6528,5632,6656,6784,6912,-1};

/*----------------------------------------------------------------
 word in dictionary
----------------------------------------------------------------*/
short int mes11[]={7040,7168,7296,7424,7552,7680,7296,3072,7296,7808,-1};

/*----------------------------------------------------------------
 Dictionary is out of memory
----------------------------------------------------------------*/
short int mes12[]={6144,6272,2176,7936,8064,8192,8320,8448,5888,8576,5888,6016,3968,-1};

/*----------------------------------------------------------------
 Speak into microphone
----------------------------------------------------------------*/
short int mes13[]={8704,8832,8960,4864,128,256,6912,9088,9216,7296,2816,9344,2560,2944,-1};

/*----------------------------------------------------------------
 Too noisy
----------------------------------------------------------------*/
short int mes14[]={9472,128,9600,-1};

/*----------------------------------------------------------------
 No matching word 
----------------------------------------------------------------*/
short int mes15[]={9728,9856,9984,-1};

/*----------------------------------------------------------------
 Too loud voice
----------------------------------------------------------------*/
short int mes16[]={128,256,9600,-1};

/*----------------------------------------------------------------
 Too quiet voice
----------------------------------------------------------------*/
short int mes17[]={128,256,10112,-1};

/*----------------------------------------------------------------
 Standby
----------------------------------------------------------------*/
short int mes18[]={10240,10368,10496,-1};

/*----------------------------------------------------------------
 Not detected yet
----------------------------------------------------------------*/
short int mes19[]={10624,10752,10880,-1};

/*----------------------------------------------------------------
 Cancel
----------------------------------------------------------------*/
short int mes20[]={11008,11136,11264,11392,11520,-1};

/*----------------------------------------------------------------
 Detecting
----------------------------------------------------------------*/
short int mes21[]={10752,10880,10496,-1};

/*----------------------------------------------------------------
 Recognition complete
----------------------------------------------------------------*/
short int mes22[]={384,512,11648,11776,-1};

/*----------------------------------------------------------------
 Strings that stores recognized word
----------------------------------------------------------------*/
short int mes[][17] = {
{ 11904,12032,4864,12160,-1} ,
{ 4864,3328,2816,3328,12288,7552,-1} ,
{ 12416,4096,12544,12672,7552,-1} ,
{ 12800,3328,12032,6016,12544,12672,7552,-1} ,
{ 7552,12544,12672,7552,-1} ,
{ 3840,7296,3840,7296,-1} ,
{ 12928,3328,4864,12544,13056,-1} ,
{ 12928,3328,13184,3328,13056,-1} ,
{ 2560,7424,7552,5632,4224,-1} ,
{ 13312,11904,3968,13440,-1} ,
{ 13184,2944,13184,2944,-1} ,
{ 8704,8192,8448,-1} ,
{ 13568,11520,13696,-1} ,
{ 13824,3328,7680,7552,-1} ,
{ 2816,3328,12288,3328,-1} ,
{ 4096,13952,12288,3328,-1} ,
{ 1408,5888,7936,11264,-1} ,
{ 12032,14080,13824,13952,-1} ,
{ 7680,3072,7680,-1} ,
{ 14208,2944,12928,3328,-1} ,
{ 4864,3328,14336,3328,-1} ,
{ 12416,5888,3072,3328,-1} ,
{ 7808,3072,14464,14592,-1} ,
{ 14336,14080,2176,2944,3840,-1} ,
{ 2560,13952,3072,2944,3840,-1} ,
{ 2688,3328,12928,3328,-1} ,
{ 14720,14720,14848,-1} ,
{ 768,896,1152,5888,-1} ,
{ 8448,14976,11264,-1} ,
{ 15104,15232,7552,-1} ,
{ 7552,12288,3328,-1} ,
{ 15360,13312,3328,-1} ,
{ 15488,12288,7552,-1} ,
{ 15616,7680,3328,-1} ,
{ 4736,3328,15104,-1} ,
{ 7936,1152,11264,-1} ,
{ 2944,12544,15104,-1} ,
{ 15744,15872,11264,896,-1} ,
{ 15360,3328,7680,7552,-1} ,
{ 14208,2944,15232,7552,16000,3328,-1} ,
{ 15744,16128,16256,8960,768,-1} ,
{ 15744,15872,16256,8960,768,-1} ,
{ 7552,16384,-1} ,
{ 7296,7808,-1} ,
{ 16512,14208,4736,-1} ,
{ 13440,14592,-1} ,
};

Those Mes arrays at the bottom might be for errors, but it doesn't seem to answer your question.
 
Last edited:
U

ulao

Member
I talked to the MESS dev and unfortunately he has dropped his involvement many years back. Does anyone here know a n64 emu developer that could assist? All we need is an emulator that boots the game to the point we can hit the z button. At that point I think we can turn on debug and watch what the game sends to the controller ports.

If we get to that point I can get the recognized HEX code from the VRU and send it on the 3rd row of buttons or analog data like the dial axis. That would be very easy to pick up in the emulator and relay to the game. At most it's a good test to get this working.
 
OP
F

Falkoner

New member
ulao said:
I talked to the MESS dev and unfortunately he has dropped his involvement many years back. Does anyone here know a n64 emu developer that could assist? All we need is an emulator that boots the game to the point we can hit the z button. At that point I think we can turn on debug and watch what the game sends to the controller ports.

If we get to that point I can get the recognized HEX code from the VRU and send it on the 3rd row of buttons or analog data like the dial axis. That would be very easy to pick up in the emulator and relay to the game. At most it's a good test to get this working.
Click to expand...

Exactly what issue are you having? If it's graphical, the game works with Rice plugin, but the other ones fail to properly render the model's textures. However, if the issue is more related to getting the emulator to properly detect the VRU, then yeah, that's why I opened up the issues on Github, I'll try to contact some of the developers directly.
 
Last edited:
You must log in or register to reply here.

Top