theWacoKid
11-29-2006, 11:14 AM
http://www.romsteady.net/blog/2006/11/quality-assurance-at-sony.html
Monday, November 27, 2006
Quality Assurance at Sony
Sam Kalman made a post on November 22nd about a bug in Genji found by Chris Kohler, and it begs for the following story to be told.
Back in April, I was interviewed for a FPQA Manager position at Sony Computer Entertainment America's San Diego office. Sony was extremely nice. They flew me down and back first-class, took me out to lunch, etc.
Everyone I met there was a consumate professional, but there was a lot of underlying tension. I signed an NDA so I can't go into specifics, but there was talk about issues that only came up on production UMD's for PSP games, major friction between test and development teams with little to no management backing for test, little to no shared technology, extremely lax "user effect" bug metrics for determining whether or not to fix something, and a variety of other fairly hefty issues, not just from a process standpoint, but a overall culture standpoint. Microsoft is known for giving QA a bit too much say in the products that are developed, but the feeling I got inside Sony was that QA was seen as nothing but a bunch of monkeys with controllers.
The straw that broke the camel's back came in the last hour of my interview. I was told that the way that Sony tests their games is that there are one or two test leads on a project starting at about six months out. At T-8 weeks, between 80 and 100 temporary testers are brought on to test the game for those eight weeks. That's it. This was done for financial reasons, and as a QA Manager, I would be expected to run test the same way. Obviously, I didn't feel that was a valid way of handling QA.
The following morning, I sent an E-mail to Sony removing myself from consideration for the position because I didn't feel that I could run test the way that they wanted me to.
At Microsoft, the stringent QA processes often strangle creativity. At Sony, the lax QA process allows creativity to squash quality. It's hard to walk a middle ground where QA and creativity work hand in hand, but it is a tightrope that this industry is going to have to learn to walk if it is going to succeed in the 21st century and beyond.
(Update: Welcome, visitors from Sony/Psygnosis and readers of the Escapist. Please don't take this as criticism of Sony, just of the practices as they were described to me. No company has QA perfected, and Sony has released some wonderful titles over the years. However, past success is not a guarantee of future success as this incident proves. Trust in Sony's ability to deliver is already shaken, not only from a consumer standpoint, but a developer standpoint as well. [Hell, I still haven't received my taxi fare reimbursement...]
First-party games are supposed to push the envelope with killer gameplay, crystal-clear graphics and first-rate quality. First-party games are supposed to sell not only the abilities of the console, but the promise of the platform.
Consider this a prod towards delivering the true promise of the platform: next-generation gaming for the masses. The masses don't like patching.)
http://blog.wired.com/games/2006/11/genjis_gamekill.html
Wednesday, 22 November 2006
Genji's Game-Killing Bug
Topic: Game|Death
So PlayStation 3 reviewers got an email this afternoon with helpful tips for Genji, straight from the development team. That's a real interesting coincidence, because I was playing Genji when I got that email and I was coming up with some tips of my own for the development team. Tips like:
When making a third-person action game, try to develop a camera system that lets the player actually see what's going on around him
Try to avoid creating puzzles with solutions that involve hitting unseen objects that lie off the boundaries of the screen (update on this inside)
You might want to avoid leaving major bugs in the game that ruin your saved progress and force you to restart the whole thing
So I wasn't too too far into Genji -- I was right at the part where you get the lady warrior Shizuka, about two hours in. And I'd been putting up with unclear mission goals for pretty much the whole time. The game's camera system isn't really good for determining where important, key pieces of scenery are in relation to you. At one point you have to smash a cart so it rolls up to a locked gate, then use it to jump over. But since the camera keeps flipping between two fixed viewpoints, it's tough to tell that the cart would even go up to the gate if you were to push it.
That's minor compared to what happened next. I had to hit two doohickeys that were on top of a roof. But I could only see them in the initial cut scene that set up what I had to do, and even then the camera angle was such that I couldn't see where they were placed in relation to the ground. Checking GameFAQs after getting really frustrated, it turned out that yes, I had to stand underneath the thingamabobs while on the ground, and swing my weapon upwards to hit them.
Not that the game ever told me that I had an upward-facing weapon strike.
Not that there was any way, at all, to see the whatsits, meaning I had to just swing around like an idiot until I randomly hit them.
(Correction: Apparently, what you're supposed to do for this puzzle is get the key, as described below, and use it to enter the building and climb to the roof to hit them. But as I said -- the key was gone!)
But even that wasn't so bad when you compare it to the fact that I didn't know what to do next, and I wasn't finding anything because there was nothing to find. So it was back to GameFAQs' board, but this time there was no solution. Just confirmation that there was supposed to be an item for me to pick up, but due to a glitch in the game it disappears after a certain point. And of course, I'd been diligent and saved my game, meaning I was completely screwed. I'd have to start the game over.
That is, until I realized that -- through sheer coincidence -- I'd copied my old Genji save to a USB thumb drive when testing that feature of PS3. So I loaded it up, worked my way back to where I was, and there was the item I needed. Had I not realized that I had an old save, I wouldn't have bothered. But this is unacceptable, anyway. Simply outrageous that something this obvious would make it through QA.
Posted by Chris Kohler 2:18 AM | Post Comment | View Comments (29) | Permalink
Sony has an attitude of we'll fix it in post. The ps3 has been out for a little over a week, and we're already up to a second firmware update. This isn't a case of a bug cropping up that was unforseen, these are cases of not giving a damn, shoving something out the door and hoping nobody notices. And oh, don't count on that scaling fix anytime soon.
PS3 scaling might not be fixed after all
By James Orry - 27/11/2006 - 11:01am GMT
Sony Computer Entertainment America is looking into scaling problems, but hasn't stated what is to be done about it.
The upshot of the problem is that owners of a TV unable to accept a 720p signal and only 480p/1080i, older HD sets mainly, are stuck with playing 720p PS3 games in only 480p resolution. With a game like Resistance: Fall of Man, which runs natively at 720p, the PS3 will default down to 480p.
Obviously the televisions are partly to blame for the problem, but if the PS3 was to scale the image like the Xbox 360 is capable of the problem would not exist. Last week Sony admitted that there was a problem and even said that it was working to provide a solution.
However, SCEA spokesperson Dave Karraker has told GameDaily that the company is "looking into the issue and haven't stated any actions that will be taken regarding it."
It could therefore be some time before a fix is available, if such a fix is ever released.
Source: GameDaily
Monday, November 27, 2006
Quality Assurance at Sony
Sam Kalman made a post on November 22nd about a bug in Genji found by Chris Kohler, and it begs for the following story to be told.
Back in April, I was interviewed for a FPQA Manager position at Sony Computer Entertainment America's San Diego office. Sony was extremely nice. They flew me down and back first-class, took me out to lunch, etc.
Everyone I met there was a consumate professional, but there was a lot of underlying tension. I signed an NDA so I can't go into specifics, but there was talk about issues that only came up on production UMD's for PSP games, major friction between test and development teams with little to no management backing for test, little to no shared technology, extremely lax "user effect" bug metrics for determining whether or not to fix something, and a variety of other fairly hefty issues, not just from a process standpoint, but a overall culture standpoint. Microsoft is known for giving QA a bit too much say in the products that are developed, but the feeling I got inside Sony was that QA was seen as nothing but a bunch of monkeys with controllers.
The straw that broke the camel's back came in the last hour of my interview. I was told that the way that Sony tests their games is that there are one or two test leads on a project starting at about six months out. At T-8 weeks, between 80 and 100 temporary testers are brought on to test the game for those eight weeks. That's it. This was done for financial reasons, and as a QA Manager, I would be expected to run test the same way. Obviously, I didn't feel that was a valid way of handling QA.
The following morning, I sent an E-mail to Sony removing myself from consideration for the position because I didn't feel that I could run test the way that they wanted me to.
At Microsoft, the stringent QA processes often strangle creativity. At Sony, the lax QA process allows creativity to squash quality. It's hard to walk a middle ground where QA and creativity work hand in hand, but it is a tightrope that this industry is going to have to learn to walk if it is going to succeed in the 21st century and beyond.
(Update: Welcome, visitors from Sony/Psygnosis and readers of the Escapist. Please don't take this as criticism of Sony, just of the practices as they were described to me. No company has QA perfected, and Sony has released some wonderful titles over the years. However, past success is not a guarantee of future success as this incident proves. Trust in Sony's ability to deliver is already shaken, not only from a consumer standpoint, but a developer standpoint as well. [Hell, I still haven't received my taxi fare reimbursement...]
First-party games are supposed to push the envelope with killer gameplay, crystal-clear graphics and first-rate quality. First-party games are supposed to sell not only the abilities of the console, but the promise of the platform.
Consider this a prod towards delivering the true promise of the platform: next-generation gaming for the masses. The masses don't like patching.)
http://blog.wired.com/games/2006/11/genjis_gamekill.html
Wednesday, 22 November 2006
Genji's Game-Killing Bug
Topic: Game|Death
So PlayStation 3 reviewers got an email this afternoon with helpful tips for Genji, straight from the development team. That's a real interesting coincidence, because I was playing Genji when I got that email and I was coming up with some tips of my own for the development team. Tips like:
When making a third-person action game, try to develop a camera system that lets the player actually see what's going on around him
Try to avoid creating puzzles with solutions that involve hitting unseen objects that lie off the boundaries of the screen (update on this inside)
You might want to avoid leaving major bugs in the game that ruin your saved progress and force you to restart the whole thing
So I wasn't too too far into Genji -- I was right at the part where you get the lady warrior Shizuka, about two hours in. And I'd been putting up with unclear mission goals for pretty much the whole time. The game's camera system isn't really good for determining where important, key pieces of scenery are in relation to you. At one point you have to smash a cart so it rolls up to a locked gate, then use it to jump over. But since the camera keeps flipping between two fixed viewpoints, it's tough to tell that the cart would even go up to the gate if you were to push it.
That's minor compared to what happened next. I had to hit two doohickeys that were on top of a roof. But I could only see them in the initial cut scene that set up what I had to do, and even then the camera angle was such that I couldn't see where they were placed in relation to the ground. Checking GameFAQs after getting really frustrated, it turned out that yes, I had to stand underneath the thingamabobs while on the ground, and swing my weapon upwards to hit them.
Not that the game ever told me that I had an upward-facing weapon strike.
Not that there was any way, at all, to see the whatsits, meaning I had to just swing around like an idiot until I randomly hit them.
(Correction: Apparently, what you're supposed to do for this puzzle is get the key, as described below, and use it to enter the building and climb to the roof to hit them. But as I said -- the key was gone!)
But even that wasn't so bad when you compare it to the fact that I didn't know what to do next, and I wasn't finding anything because there was nothing to find. So it was back to GameFAQs' board, but this time there was no solution. Just confirmation that there was supposed to be an item for me to pick up, but due to a glitch in the game it disappears after a certain point. And of course, I'd been diligent and saved my game, meaning I was completely screwed. I'd have to start the game over.
That is, until I realized that -- through sheer coincidence -- I'd copied my old Genji save to a USB thumb drive when testing that feature of PS3. So I loaded it up, worked my way back to where I was, and there was the item I needed. Had I not realized that I had an old save, I wouldn't have bothered. But this is unacceptable, anyway. Simply outrageous that something this obvious would make it through QA.
Posted by Chris Kohler 2:18 AM | Post Comment | View Comments (29) | Permalink
Sony has an attitude of we'll fix it in post. The ps3 has been out for a little over a week, and we're already up to a second firmware update. This isn't a case of a bug cropping up that was unforseen, these are cases of not giving a damn, shoving something out the door and hoping nobody notices. And oh, don't count on that scaling fix anytime soon.
PS3 scaling might not be fixed after all
By James Orry - 27/11/2006 - 11:01am GMT
Sony Computer Entertainment America is looking into scaling problems, but hasn't stated what is to be done about it.
The upshot of the problem is that owners of a TV unable to accept a 720p signal and only 480p/1080i, older HD sets mainly, are stuck with playing 720p PS3 games in only 480p resolution. With a game like Resistance: Fall of Man, which runs natively at 720p, the PS3 will default down to 480p.
Obviously the televisions are partly to blame for the problem, but if the PS3 was to scale the image like the Xbox 360 is capable of the problem would not exist. Last week Sony admitted that there was a problem and even said that it was working to provide a solution.
However, SCEA spokesperson Dave Karraker has told GameDaily that the company is "looking into the issue and haven't stated any actions that will be taken regarding it."
It could therefore be some time before a fix is available, if such a fix is ever released.
Source: GameDaily