The guidelines and rules tester adhere to when completing tests on Rainforest.
Rainforest QA has a set of testing rules every tester is required to follow when testing your application. As a Rainforest client who writes test cases or reviews test results on a regular basis, it’s important you understand the rules our testing community must adhere to. The rules are as follows:
- Rule 1: Pay Close Attention to the Whole Step
- Rule 2: Pay Close Attention to Quotes & Placeholders
- Rule 3: Categorize & Report Pop-Ups & Errors Appropriately
These rules exist to ensure consistent, high-quality results for you and others. Please read on for a more thorough breakdown of each rule and how it applies to you.
A Note on the Suggest Improvement Feature
You’ll see the Suggest Improvement (SI) feature referenced throughout our testing rules.
Testers may use Suggest Improvement only if the answer to a question if yes — think of it as a “Yes, and … “. They might want to leave you a tip about how your instruction could be phrased better. We encourage testers to use the SI feature in addition to answering the step with “YES”.
Testers can use SI to report situations falling under the 5 categories listed below:
- This step can be broken into multiple steps: you may see this if you are asking them to do too many things in one step, and this is confusing;
- These instructions are hard to understand: you may see this if the testers bumped into some jargon they don’t quite understand, or phrasing that could be clearer;
- The action you want me to do is not fully described: you may see this if your instructions omit certain actions that are too far of a leap for testers to make, or you maybe forgot to add something to make them complete;
- There was an unexpected pop-up in this step: if they bump into a pop-up you didn’t account for in instructions, they think is benign (see Rule 3) but think you should know about it anyway;
- I see an additional problem that is not asked as part of the question: if they notice something else about your app, e.g. a typo in a part not mentioned in instructions.
While we recommend testers use this in certain situations, it is never mandatory. If the tester chooses not to leave a suggestion, they must follow your instruction to the best of their ability and answer “YES” or “NO” accordingly.
Should they choose to use it, we ask that their report includes a polite, helpful comment explaining how the situation could be improved. Unless the tester’s comment was rude or unhelpful, you should never give the tester a poor rating (e.g. 1-2 stars) for using this feature at an appropriate time.
Rule 1: Pay Close Attention to the Whole Step
In this rule, we set the expectation that you don’t want the tester to rush through instructions or jump to conclusions. The tester must read both the step instruction and question (e.g. “Hit the Login button. Were you successfully logged in?” ) before performing any actions. Otherwise, they are at risk of providing inaccurate results, and their work may be flagged for further review by Rainforest QA.
What this means for you and your test suite
- The instructions you provide the tester must be clearly written, and must be accompanied by a single YES/NO question per step. See our section for test writing best practices for more detail.
- Testers are only allowed to answer YES if they fully understand the instructions and the question provided, they followed those instructions exactly, and the answer to the question is YES. Otherwise, the tester must click NO. Unless the tester’s bug report was rude or unhelpful, you should not give the tester a poor rating if they submitted unsatisfactory results that can be traced to poorly written instructions.
- If you want the tester to remember some specific information for a future step, you must make your intention clear in the step instructions where the tester first encounters the information. For example, “Click the VIEW DETAILS button. Is the WEIGHT & DIMENSIONS section now displayed on the page? (**please take note of the information you see under the WEIGHT & DIMENSIONS section. You will need to recall this information in a future step).”
- You should never write instructions in a way that forces the tester to answer Yes and agree to skip several steps without executing them in order to reach a later one (e.g. “Answer YES to questions 5-9 and start working from 10”). Rainforest advises testers to immediately report a job with NO should they encounter a step with instructions of this nature. Unless the tester’s comment was rude or unhelpful, you should never give the tester a poor rating (e.g. 1-2 stars) for answering NO to this type of instruction.
- Note: it is acceptable to request a tester do this from one step to another, but not for many. E.g. “You may or may not encounter an pop-up on this step. Continue to the next step if you do.” is an example of an acceptable step instruction.
Rule 2: Pay Close Attention to Quotes & Placeholders
In this rule, we set expectations around your use of quotes (e.g. “text”) and placeholders (i.e. _ ) within your test instructions.
We tell testers that (with minor exceptions, e.g. capitalization) if you use quotes in your instructions, it means that you want the tester to check that the text or values within quotes appears exactly as it appears in your application. For example, “Type the book title “Great Gatsby” into the search bar and hit enter on your keyboard. Are you brought to a page outlining the synopsis for the book?”
We also explain to testers that when you write test instructions, you are allowed to use an underscore (i.e. _ ) as a placeholder to stand in for variable text that will populate in your application. For example, Type “football cleats” in the search bar and hit enter on your keyboard. Are you directed to a page that says “_ results for football cleats”?
What this means for you & your test suite
- If all relevant aspects of quoted text match (i.e. spelling, words, or punctuation) AND testers get the expected outcome of the step question, they must answer YES. Otherwise, they must answer NO.
- We instruct testers not to take capitalization differences into account, even when quotes are used. You should never give a tester a poor rating for not failing a step over capitalization differences.
- Exact matches are important to most of our clients, but typos get the best of us. This is how testers handle various kinds of typos when it comes to quotes:
- Mixed quotes: If you include quotes ‘like this”, testers are instructed to assume that your intent was to quote the text and that they should follow the quotes rule accordingly. This ‘mixed quotes” scenario is also a time where we encourage testers to use the SI category “I see an additional problem that is not asked as part of the question” and report the typo to you to reduce confusion
- Incomplete quotes: If your instructions include incomplete quotes (“like this), testers are expected to ignore matching since the quotes are incomplete and they can’t guess where the quote might start or end, but leave a comment using the SI category “These instructions are hard to understand” to point out the incomplete quote in case an exact match was your intent.
- Typos in quoted texted: If testers notice a typo within the quoted portion of your instructions, testers are expected to assume the typo is intended, and answer Yes or No depending on whether there’s an exact match between the quoted typo and what they see within the app. If you make a typo in the unquoted portion of your instructions, we encourage testers to report this to you using the SI category “I see an additional problem that is not asked as part of the question.”
How to use quotes in your Instructions
- Only use quotes if it’s absolutely necessary for your step instruction, and avoid using them if they’re not critical to the outcome of your test.
- We recommend that you only use double quotes (“like this”).
- Don't use CAPS or asterisks if what you require an exact match. If getting an exact match is your intention, quotes are your friend.
- If you choose to use quotes, make sure you are using them in a situation where an exact match is possible. Here’s an example of when it’s appropriate to use quotes vs. when it is not:
- Appropriate use of quotes: “Look for the “save” button. Does clicking on this button enable a pop-up asking to confirm whether or not you’d like to save?”where the button displayed in your application actually has the text “save” on it.
- Inappropriate use of quotes: “Look for the “save” button. Does clicking on this button enable a pop-up asking to confirm whether or not you’d like to save?”where the save button displayed in your application does not have the text “save” written on it, but is instead another image or graphic (e.g. floppy disk icon). In situations like these, we advise testers to answer “Yes” if it’s obvious to them what you mean and report the inappropriate use of quotes to you using the SI feature.
How to use placeholders (i.e. _ ) in your Instructions
- You can use the _ placeholder inside or outside of quotes, as well as for an unlimited amount of letters, words, numbers, special characters, etc.
- You may use a single underscore (like _ ) or multiple underscores stringed together (like ___ ) to represent one single placeholder, to ensure this is visible.
- We also recommend you don't use placeholders other than _ but you may use other text or symbol(s) as placeholders if absolutely necessary. If you choose to do this, make the intention clear in your instructions before you use it. For example, “In this test, instead of the __ placeholder, we will use X as a placeholder, where X is the number of items displayed on a page.”. If you don't, testers are expected to answer with “No” if they encounter instructions using placeholders other than _.
Rule 3: Categorize & Report Pop-ups & Errors Appropriately
In this rule, we set expectations around how testers should handle certain pop-ups (including errors that come in pop-up format). Pop-ups can mean different things to different apps or sites, so it's important to know how testers will interpret them.
Rainforest defines a pop-up as anything that suddenly appears (text and/or window) and you don’t need to hover over it in order to see it.
We categorize pop-ups into 3 groups:
Group 1: Pop-up which indicates a Bug
This is a pop-up that includes an ERROR/CRASH notification. Basically, it indicates that something went wrong. Testers are expected to report these pop-ups with “No” unless specifically told otherwise in your instructions.
Group 2: Pop-up which doesn't indicate a bug
This is a pop-up that indicates an expected system/browser/application behavior (e.g. a side chat-box, marketing ad, or email subscription offer). Testers are expected to ignore these pop-ups and move on with your test.
Group 3: Pop-up which may indicate a bug, but testers aren't sure if this is the case
If the pop-up doesn’t fall into one of the two other categories (i.e. it’s not an ERROR/CRASH pop-up, and it’s not an expected behavior by the system/browser/application), and the tester thinks there is a slight chance that this pop-up may indicate a bug, then they are expected to take the most reasonable action to close it, notify you using the SI feature of they choose, and move on with their test if possible.
What this means for you & your test suite
Given the thorough guidelines we provide around pop-ups, there’s usually no need for you to include specific instructions for how testers should interact with pop-ups. However, if you have specific needs and instructions that contradict the main rule, especially if your instructions instruct testers to ignore errors or all pop-ups, you must be very specific in the language you use.
- If you instruct a tester to ignore all pop-ups, they are expected to interpret this as an instruction to ignore all pop-ups, except ERROR/CRASH notifications.
- If you instruct a tester to ignore all pop-ups including error pop-ups, they are expected to ignore all pop-ups, including ERROR/CRASH notifications. Be very careful with this - a poorly set up or buggy testing environment can lead to frequent failures and confusion, since testers are on the lookout for bugs. Even so, if a tester encounters an ERROR/CRASH notification they think you should know about despite your warnings, they are allowed to alert you using the SI category “There was an unexpected popup in this step”, and then continue with the test if possible.
- If you want the tester to ignore a specific type of ERROR/CRASH pop-up, then you must clearly state this in the instructions along with a description of the specific ERROR/CRASH pop-up they should ignore. Otherwise, testers will report the pop-up with “NO”.
There you have it!
For further details about how testers will handle pop-ups that appear during testing, please see our documentation on how to test with pop-ups.
If you have any questions about the rules testers are supposed to follow, please let our team know at firstname.lastname@example.org!