Getting feedback -- how many reviewers do you need?

I'd like to draw on the experience of this group.
The Beta of our first training lesson is complete.  We did a careful internal review with a group of subject experts who offered detailed feedback. 
Next, we offered it to a variety of managers and their staffs -- the kind of people who would either take the training or send their staff to take it. 
Of that group (between 500-1K people), we had 32 people offer to review the lesson and send in comments of their own free will. 
However, of the 32, only five brave individuals have sent in feedback as of the deadline.  Life is busy, etc.
I know that in UX design and webpage design, you can get very good usability testing even with a small group.  A group of 5-10 representative people will catch almost all of the problems you are trying to find in testing.
What is your experience with reviewing Captivate training? 
In this case (unlike usability), we are not asking people to accomplish specific tasks, but to simply take the lesson and send their feedback on both style and content.
Is my small group of five (somewhat representative users) enough?  Or do I need to seek more?  And, in your professional opinion, how many more will catch what we need to change?
Thanks in advance!

My advice is to run with the sample reviews you currently have, make any changes that were highlighted as necessary, fix any bugs that came to light, and then get the course out in the wild as quickly as possible.
There's this perception in some manager's (and Captivate developer's) minds that you can't release the course "until it's perfect".  This is a mistake, because the course will never be perfect according to somebody, and you will never be able to get it released.
So waiting for everyone to be happy with your course will result in huge delays, by which time someone else will not be happy, or the business process will have changed, or the management team will have changed, or...you get the picture.
So cover off any glaring errors found in the review, and get it out there ASAP.  You can fix anything else that comes to light in 6 or 12 month's time when it's time to do the inevitable updates to content.  If there's some other showstopper that's "discovered" before then, do the change and publish an updated version.
These constant changes are what make using CD ROMs to deliver your course a bad proposition.  Totally online is the way to go.  You can put up an update overnite and nobody is the wiser.

Similar Messages

  • How many problems do you need to have with a mac book pro to get it replaced i have had 6 problems and they say they have fixed everything and somthing new happends

    how many problems do you need to have with a mac book pro to get it replaced i have had 6 problems and they say they have fixed everything and somthing new happends
    I have had thehad the:
    ram replaced
    Battery
    Audio
    Trackpad
    os operating system
    fans
    And they still won't replace the laptop for me worst buy ever but i have had a imac for 2 months and nothing at all wrong .

    You could try calling Apple and ask for Customer Relations.
    From Readers Digest-February, 2005
    How to Complain
          You call customer service to complain about a product, and you hang up angrier than when you started. That’s customer rage, a feeling experienced by millions of people with a major complaint, says Scott Broetzmann, president an American firm that tells companies how to offer the best customer service. His secrets to getting good service:
    Have a goal
    If you want your product repaired, say so. Want an apology? Speak up.
    Keep it short
    Focus on one problem, and be succinct.
    Stick with it
    You have to invest the time it takes. Don’t get what you want? Ask for a supervisor.
    Skip ultimatums
    Don’t threaten not to do business with them again. Why should they help you if you won’t buy from them in the future?
    Plead your case
    Many companies have information such as how much money you’ve spent with them and how often you complain. If you’re a good customer, they may be more willing to help.
    Be nice
    You’re unlikely to get what you want if you’re rude.
    Good luck.

  • Calculating how many managed servers you need.

    Hi,
    Does anyone know a BEA webpage were it tells you, on the lines of :
    If you have 5000 customer hitting your weblogic cluster you only need 1 managed server. If you have 30,000 you need two managed servers. If you have 500,000 you need three.
    Does anyone know of a performance page like this?
    Regards,
    Alistair.

    There is no such performance numbers from BEA. I think you need to rely on capacity planning process to estimate number of server based on your workload and performance requirements.
    http://dev2dev.bea.com/pub/a/2002/11/Hazarika.html
    http://edocs.bea.com/wls/docs81/capplan/capfact.html#1009720
    There are some benchmark results for WebLogic are available at
    http://edocs.bea.com/wls/docs81/capplan/capbase.html#1054891
    http://www.spec.org/jAppServer2004/results/jAppServer2004.html
    -Jayesh

  • How many backups do you need to have it wrong?

    Everythime you put the plug into your computer the ipad is synconising, updating and backing up, loads and loads of precious time there spent in (you think) your own interest: you never know something can go wrong.
    However when you really need that backup there you go it is a failed one.
    Everything started when I found that the new ios5 was now available, I have an Ipad 1.
    Ok, I backup and synchronise, etc.etc. The day comes I am ready to update the ios5. I begin with a nice "backup failes error -50000". ok can happen,
    I again rebackup, all seems well and then "backup failed error -50000". By the way itunes had already eaten 42 gigs out of my hardrive.
    Lets try from a different computer then, I have up to five.
    Again, backup, synchronise, loads of time spent, where nearly there 99%. No Apps working, the wifi is off. And again "backup failed -50000".
    Ok I have a previous back up which went well, I can always use that, I (wrongly)  think.
    Ok go frward IOS5. Of course it wants to backup again, I say yes, why not, I've spent already four hours like this. It fails and goes on putting the IOS5.
    What a surprise when I find that the blooming think doesn't work at all, I actually attach it to restore the old backup and it replies "failed backup": excuse me? what?
    I push the restore button and I find myself with the IPAD asking to attach it to the itunes while the itunes is sinchronising to whom I do not know, because te ipad is not respondng!
    Then I find the bar (the one that tell you the ipads capacity, which oscilates between colours.
    I was very happy till now, but I can't believe that after a year of backups and total confidence in the machine, I have been wiped out of my own memory, everything..........
    So what do I do now? do I tell th F....Device it is new? do I restore form the icloud I do not have or do I believe n the third choice which does not work as the other two: restore from Itunes backup. Restore what? Recover what?
    Very PO sorry dissapointed, this IOS5 is a FFraud.

    You could try calling Apple and ask for Customer Relations.
    From Readers Digest-February, 2005
    How to Complain
          You call customer service to complain about a product, and you hang up angrier than when you started. That’s customer rage, a feeling experienced by millions of people with a major complaint, says Scott Broetzmann, president an American firm that tells companies how to offer the best customer service. His secrets to getting good service:
    Have a goal
    If you want your product repaired, say so. Want an apology? Speak up.
    Keep it short
    Focus on one problem, and be succinct.
    Stick with it
    You have to invest the time it takes. Don’t get what you want? Ask for a supervisor.
    Skip ultimatums
    Don’t threaten not to do business with them again. Why should they help you if you won’t buy from them in the future?
    Plead your case
    Many companies have information such as how much money you’ve spent with them and how often you complain. If you’re a good customer, they may be more willing to help.
    Be nice
    You’re unlikely to get what you want if you’re rude.
    Good luck.

  • How many times do you need to reinstall, reset, uninstall?

    and the thing still won't function properly. i mean who really cares. it doesn't have a good sound anyway, and it will probably get lost in the first 30 days.
    Message was edited by: mainflash

    Only once.
    What is the specific issue that is occurring?
    Most AirPlay and HomeSharing issues are the result of network problems.

  • How many attempt do you get in entering your restrictions code?

    how many attempt do you get in entering your restrictions code?

    I think its 3 or 5.
    If you forgot you're restriction code you will need to restore the device

  • How many times do you get a new iPhone because of manufacturing  defects, before you can ask for an newer model?

    How many times do you get a new iPhone because of manufacturing  defects, before you can ask for an newer model?

    yofromoslo wrote:
    How many times do you get a new iPhone because of manufacturing  defects, before you can ask for an newer model?
    You don't.  Apple's warranty provides for replacing your device with an identical model (same model, carrier, capacity & color).
    And you aren't getting a new iPhone.  You'll be getting a remanufactured iPhone, which has gone through a more thorough inspection than any new iPhone ever does.
    If you want a newer model, then trade-in or privately sell your current device and use the proceeds towards purchasing another device.

  • How many licenses do you get when you purchase the Adobe Creative suite?

    How many licenses do you get when you purchase the Adobe Creative suite? I'd like to install it on one more computer but do not know if this is allowed.forumnotifierAmy_WongDebbieWoojin

    A single user license (one license) purchase allows for you to install and activate on two machines. 

  • When I went and connected  phone to my laptop and hit restore and it got rid of everything my photos my contacts everything and I don't know how to get the stuff back and if you need to know I have an IPhone4S

    When I went and connected my phone to my laptop and hit restore and it got rid of everything my photos my contacts everything and I don't know how to get the stuff back and if you need to know I have an IPhone4S

    I take it you were replacing an older iPhone and this was the first time you plugged it in? If so, that's what happens. It restored the backup of your old phone. A backup of the current phone is not made in those circumstances.
    If that's not what happened, you need to clarify.

  • How many downloads do you get?

    I have two computers. If I buy illustator, how many downloads do you get? Can I download it to both of my computers?

    Yes, you can load it on two computers as long as they are not in use concurrently.

  • How many amps do you get

    how many amps do you get for guitar

    Same as you get in Logic:
    http://www.apple.com/logicpro/plugins-and-sounds/

  • How many FLVs before you get problems?

    I have built a small project in Captivate 4 (30 screens with 15 FLV files ranging in duration from 30 - 60 ; FLV sizes are between 3 and 6 mb).  I am encountering the following issue:
    Course plays fine when FLVs are unsynched
    FLVs "stutter" randomly when synched to project
    The video was shot at 29.97 FPS but I have re-exported it at 30 FPS.  I also set preload to 100% and strangely this reduced the severity of the issue but did not fix it (I say strange as the FLVs should not be affected by the pre-loader)
    Has anyone else experienced this issue?  My questions are:
    How can I synch FLVs without stutter during playback
    Is there a limt on the number of FLVs that can be included in an individual project?
    Is there a limit on the total duration of FLV footage that can be included in a project?
    Much zen and good wishes to all who reply (and potential consulting gigs!!!)
    Cheers
    Mark

    yofromoslo wrote:
    How many times do you get a new iPhone because of manufacturing  defects, before you can ask for an newer model?
    You don't.  Apple's warranty provides for replacing your device with an identical model (same model, carrier, capacity & color).
    And you aren't getting a new iPhone.  You'll be getting a remanufactured iPhone, which has gone through a more thorough inspection than any new iPhone ever does.
    If you want a newer model, then trade-in or privately sell your current device and use the proceeds towards purchasing another device.

  • How many times do you have to click the sleep/wake button to get the light brighter? (on iPod touch)

    How many times do you have to click the sleep/wake button (top button) to get the light/screen brighter? (on iPod touch)

    That is not haow you change brigntness. Go to Settings>Brightness to adjust screen brightness.

  • CS Review - How many reviewers can I invite?

    Hi all, I have been reserching about this type of sevices-interface for the last 3 weeks, since in my new job there are paying a lot of money to a company called LimeHouse, which has an online application similar to cs review. I am planning to present this product to my supervisor next week, so I need some info about CS Review (which I think is magical for publishing works since is integrated to InDesign). My first question is...
    How many reviewers can I invite?
    I know you need an Adobe ID account, which is free. I know if I sign in for the PLUS services I get up to 20 participant, but this means I can't invite to review my document to more than 20 people? Most of my work has to be uploaded to the web since I work for a goverment office and the public has to participate in the review as well as teammates.
    I hope I was clear.
    Thanks
    Jorge.

    Hi
    There is no limit to the number of reviewers you can invite to your reviews.  With a review open in the web client, click "Share File", enter their email address, and click send. They will receive an email invitation to the review. If there is an Adobe ID associated with their email they will be taken to a screen where they can login.  If they do not have an Adobe ID, they will be prompted to sign up for a Adobe ID/free Acrobat.com account. This process for this sign up/sign in ensures that there is verified identity with those people you collaborate with and login/password to access to reviews.
    Another tip- You can see your invitees status by clicking on their name in the collaborator bar at the bottom of the review.
    Where did you read about 20 participants? If this posted somewhere I'd like get it corrected. 
    On a side note- I'm curious as to how many participants do people typically invite their reviews?
    Hope this helps!
    Mike

  • Q: How many Users do you serve w. your OES11-Infrastructure?

    Reason for this - at first glance - stupid- question is the following:
    We migrated (or Novell forced us to) from our low power (two HL DL360 G3 and 2 DL360 g4 one G5) 6.5 Cluster to a very juicy (6xDL360 G8, tons of proccessors) OES 11 cluster. We serve
    approx. 1600 - 1700 Clients. Today everything stopped some of the Cluster Nodes didn't respond. This never happend in our old Cluster!
    The reason was or IS by "Enterprise" Design:
    DMESG: ipv4: Neighbour table overflow. !!!! Are we the only ones in the (small Novell) World who serves more than 1000 Clients???? WHY is there a limitation in the IP Cluster stack???
    Did one EVER tested that. Should we better migrate to M$???
    ip neigh show | wc -l --> shows 1021 entrys
    sysctl net.ipv4.neigh.default.gc_thresh3
    net.ipv4.neigh.default.gc_thresh3 = 1024
    Means one cluster Node can handle 1024 connections. What if on one Node runs the DNS server? Only 1020 of 1600 Clients get an answer! Cool enterprise solution boys!
    VERY angry!

    On 03/25/2014 04:56 AM, jottschi wrote:
    >
    > Reason for this - at first glance - stupid- question is the following:
    > We migrated (or Novell forced us to) from our low power (two HL DL360 G3
    > and 2 DL360 g4 one G5) 6.5 Cluster to a very juicy (6xDL360 G8, tons of
    > proccessors) OES 11 cluster. We serve
    > approx. 1600 - 1700 Clients. Today everything stopped some of the
    > Cluster Nodes didn't respond. This never happend in our old Cluster!
    > The reason was or IS by "Enterprise" Design:
    > DMESG: ipv4: Neighbour table overflow. !!!! Are we the only ones in the
    > (small Novell) World who serves more than 1000 Clients???? WHY is there
    > a limitation in the IP Cluster stack???
    I'm sorry to hear about the lack of availability of the cluster nodes. I
    am sure it is frustrating to have a system configured to be
    highly-reliable (via clustering) to fail due to default settings. With
    that said, this is a default Linux setting, and you're welcome to change it.
    > Did one EVER tested that. Should we better migrate to M$???
    I do not understand what you're asking.... did anybody test this on large
    broadcast domains?
    > ip neigh show | wc -l --> shows 1021 entrys
    Wow, well that's quite a few. I think this number indicates something you
    do not realize, though. While something like 'ss' or 'netstat' will tell
    you how many layer three (IP) or four (TCP/UDP) connections your system
    has, that number is not reflected by 'ip neigh', so usually the number of
    clients your system has is limited in the list of neighbors by quite a
    bit. For example, if I make a connection on my laptop (openSUSE, but same
    default for my neighboring table) to Google, I can see that I have a fair
    number of network connections total as shown:
    Code:
    me@mybox:~/Desktop> /usr/sbin/ss -planeto | grep -c ESTAB
    61
    but that's only tangentially related to the number of neighbors my laptop
    sees:
    Code:
    me@mybox:~/Desktop> ip neigh
    192.168.1.20 dev eth0 lladdr 00:1d:09:03:54:02 STALE
    192.168.1.254 dev eth0 lladdr 00:1e:2a:74:66:35 STALE
    192.168.1.1 dev eth0 lladdr f8:8f:ca:40:7a:1c REACHABLE
    192.168.255.50 dev br8 lladdr ac:d3:58:ae:8e:ac STALE
    How can that be? I have at least eight connections to unique remote
    machines, so shouldn't I have at least eight listings in 'ip neigh'
    output? The answer, of course, is 'no' because what you see from 'ip
    neigh' are the cached addresses found via ARP, and ARP only matters within
    the current broadcast domain, meaning on your local network before you hit
    any routers. Even though I'm connected out to eight machines, all my
    local routing table needs to care about (despite being VPN'd to a few
    places and therefore having a few extra networks considered semi-local)
    are the server, router, another router, and printer. Also, this number,
    as you can see above, has entries become stale pretty quickly. I think
    that time period is, again by default, something like thirty seconds,
    since there is no guarantee from one second to the next that a given
    server, workstation, or other node on the local network will still be
    there. After something is stale the system just uses ARP again to see
    what's still out there.
    > sysctl net.ipv4.neigh.default.gc_thresh3
    > net.ipv4.neigh.default.gc_thresh3 = 1024
    Yes, this is a default for the Linux kernel in general. You're welcome to
    tune it using sysctl, or probably via Yast so that it is stored to be used
    after rebooting (and for the nice simple UI, if you're into that kind of
    thing).
    > Means one cluster Node can handle 1024 connections. What if on one Node
    > runs the DNS server? Only 1020 of 1600 Clients get an answer! Cool
    > enterprise solution boys!
    No, it means that a given box, by default, can handle 1024 connections
    from the local network. A typical Class C network, for example, only has
    254 possible nodes, so only 253 connections. I've seen enterprises do
    some supernetting to get things like 1022 nodes possible in a network, and
    sometimes even fill that, but that's it. I have been on networks that had
    addresses handed out from a Class B-like range (172.16.x.x) or even a
    class A range (10.x.x.x) but those networks never had 65k or 10M boxes on
    them because when you get too far beyond a thousand nodes you typically
    suffer a bit of slowness. In your case you not only have more than 1024
    nodes, but all of them are talking to the same server at the same time on
    that same network.
    Can Linux handle more? Sure, you're dealing with the same OS that runs
    most supercomputers in the world, many of them being large clusters of
    computers all working together in a single network, so obviously the
    potential is there; however, there are reasons that one does not allow the
    kernel to chew up all memory for a setting like "how many neighbors do you
    have cached right now" when, for 99.9999% of the world, that max number
    is around 200 since this specifically deals with neighbors on the same
    segment or broadcast domain, not neighbors meaning anything reachable via
    IP. Doing otherwise, for example setting it to 1,000,000, means that
    somebody who was clever could more-easily waste your system's memory via a
    denial of service (DoS) attack by flooding the network with bogus ARP
    entries. Every one takes some memory, and every one needs to be managed
    by the kernel until it is expired and removed, and again those networks
    needing more than 1024 to be remembered within any single thirty-second
    period are pretty rare. Your network must be pretty awesome to handle
    that number of concurrent systems in the same logical network without
    degradation.
    Regarding running a DNS server, that is a good possibility as a service
    that would experience this problem assuming all of the clients were on the
    same network as the DNS server itself. While possible, the number of
    networks where I've seen that has been pretty small overall since usually
    those machines (servers) end up in a DMZ somewhere to prevent
    unauthorized, or at least unaudited, access.
    > VERY angry!
    You're right in that this limitation should be documented, and made quite
    obvious wherever it is documented. The migration guide would make sense,
    as that would be a place where a lot of assumptions about defaults would
    be questioned due to the OS change. If you have a specific page in the
    documentation where you read through and expected limitations to be called
    out, please either submit feedback, perhaps linking to this thread to
    provide the background that you experienced, or post a link here an I'll
    do the same.
    If there are any other defaults that you feel should be changed,
    particularly for your environment, sharing those may help as well.
    In the meantime, you can change your current settings by running the
    following as 'root' if you have not done so already:
    Code:
    cp /etc/sysctl.conf /etc/sysctl.conf-`date +%s`
    cat << EOM >> /etc/sysctl.conf
    # Setup higher threshold for arp
    net.ipv4.neigh.default.gc_thresh3 = 4096
    net.ipv4.neigh.default.gc_thresh2 = 2048
    net.ipv4.neigh.default.gc_thresh1 = 1024
    EOM
    To apply the changes either run 'sysctl -p' (from memory I think that's
    the right command) or else reboot the node, then perform again on other nodes.
    Good luck.
    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...

Maybe you are looking for

  • Runtime error while putting negative value in to table control field

    Hi Guys,               I am working on a Table Control which has Currency and quantity fields.In PBO when the program is trying put a negative value then It is going into a runtime error. The Error description is, Runtime errors         DYNPRO_FIELD_

  • Dual display problems when using keynote, powerpoint etc..

    I can run my macbook in mirroring mode and extended desktop no probs but as soon as I run programs such as powerpoint, keynote or presenter software it will run the slides on the MacBook screen and put the control/operators screen on the projected sc

  • Problem in compiling a java file

    hello... i placed all my java file in the folder C:\nmt and i run the following command to compile it... javac -sourcepath C:\nmt testabc.java now it showing an error.. cannot read testabc.java javac -sourcepath C:\nmt -d C:\nmt testabc.java to produ

  • Need a copy of the end user licence as would like to put Firefox on a network

    We are looking to changing the internet browser on our desktop image, for this I need a copy of the EULA to ensure that we comply with FAST. FAST does not usually allow for Freeware to be installed and compliant.

  • Subtotal in trailer on conditional break (PDF Form)

    A row put by conditional breaks as trailer ( with condition "Items[-1].VPROC ne Items.VPROC" for example). How to calculate subtotal on some field in this row? Any example?