FAQs for researchers

Purpose of FindingFive

What can I do at FindingFive?

FindingFive is a complete, standalone, one-stop platform for running online behavioral studies. You can quickly set up an online behavioral study without any prior experience in designing a web app. You have customized control over participant recruitment that makes sense. To make things even better, as soon as a study session is finished, participant data are organized in a sensible format for immediate download.

You can also use FindingFive with Amazon Mechanical Turk. You can still create and manage your studies entirely on FindingFive, but the studies are published on Amazon Mechanical Turk instead of FindingFive. Check out Launching a study on Mechanical Turk for details on how to set up your MTurk account on FindingFive.

Why should I use FindingFive instead of Amazon Mechanical Turk?

FindingFive streamlines the research process by allowing researchers to create professional-looking behavioral studies, control the recruitment of participants, run studies and collect data directly on FindingFive. These features are designed specifically for researchers and not available or not easily achievable with Mechanical Turk alone.

To make it easier for researchers to “make” online studies, FindingFive introduces a new grammar, named Study Specification Grammar (SSG), which does not involve much programming at all. Instead, it spells out both the content and logic of a study in intuitive terms that are already familiar to behavioral researchers. We have written a lot of documentation to help you get started using the SSG - check out the Crash Course and the Study Specification Grammar Reference.

Our ultimate goal is to build a high-quality participant community. FindingFive aims to build a community of participants who are genuinely interested in behavioral research. These participants will be given full privacy protection, and at the same time, are encouraged to interact with researchers regarding the studies that they have contributed to. We feel it is important to provide the feedback to study participants on how their data mattered, and led to scientific discoveries.

As we are working towards this goal, it is possible to use FindingFive with Amazon Mechanical Turk! You can reap the benefits of

  • Creating your studies using the FindingFive SSG
  • Using recruitment controls to perform tasks such as preventing Turk workers from participating in the same study twice
  • Hosting your study on FindingFive for free - All studies on FindingFive are also hosted on FindingFive, completely freeing researchers of the mundane tasks of setting up a server and obtaining SSL licenses that would be required to run an external study on Mechanical Turk.

Check out Launching a study on Mechanical Turk for details on how to set up your MTurk account on FindingFive.

I am a JavaScript expert and already know other platforms very well. Why should I invest time in using FindingFive?

Because coding up a complete experiment from scratch is a significant time commitment, and taking care of the nitty-gritty details can be annoying. For simple studies that do not require a lot of customizations, FindingFive can save you a lot of time! Check out the Crash Course to see how straightforward it is to create a study.

Logistics

I am a student running studies on behalf of my lab. How should I sign up?

Regardless of whether you are a graduate student dedicated to research or an undergraduate research assistant, we encourage you to sign up as a researcher and as yourself on FindingFive. To show that you are conducting research on behalf of your lab (or research group), go to your profile and add the corresponding affiliation information. When your study is published, participants will see it as coming from your lab rather than you personally.

How do I create a study?

Studies are created by specifying the structure of your experiment using our Study Specification Grammar. The grammar allows for various types of randomiziation, blocking control, presentation of different types of stimuli, and collection of many types of participant responses. In the very near future, the grammar will also support interactive presentation of stimuli, such as the mechanism of a self-paced reading study. Check out the Crash Course, and keep the Study Specification Grammar Reference open for reference.

How do I pay my participants?

For studies launched on Mechanical Turk, payment is processed by Amazon. You should make sure your AWS account balance has enough funds to cover the cost of your session.

For studies launched directly on FindingFive, we are now conducting a closed beta test that supports paying participants directly via PayPal. If you are interested, please contact our researcher support team researcher.help@findingfive.com to enable this direct payment feature for your account. See Participation Reward and Direct Payment for more details.

You are free to arrange payments on your own as well. Please provide sufficient instructions to your participants on how they will get paid if your study promises a cash reward.

Can I reward participants with course credits?

Of course! You can launch a study on FindingFive without any cash compensation. You can specify what course credits will be granted to your participants in the description field of your study.

Study design

What types of stimuli can I use in my study?

We currently allow you to use Stimuli in the form of text, images, audio, and video files. We make it simple for you to upload your stimulus files into FindingFive’s system (with a temporary restriction of 2MB or smaller). In the future, we plan to lift this size restriction as much as possible.

What types of participant data are recorded?

The types of participant data recorded during a study session are dependent on the types of Responses used in a study. For example, if a trial features a two-alternative forced choice response, it automatically records a participant’s choice, the reaction time, and whether it is a correct response or not (if the correct answer is given). In general, recording of participant data is automatically handled by our platform for researchers.

If my participants aren’t in the lab, how do I make sure they’re paying attention during my study?

The study specification grammar offers a built-in catch trial feature. This allows you to set up your own test to make sure that participants are paying attention and on-track with your study. For example, let’s say your study asks participants to listen to lists of words, presented as audio stimuli. You could set up a catch trial every 5 trials, where participants are asked to type the word that they just heard on the previous trial. This allows you to check whether the participant correctly heard your audio stimulus, and whether their attention wavered at any point during the study. Read more about it in the Procedures documentation.

The study specification grammar isn’t sophisticated enough for my study. What can I do?

We are continuously improving the grammar to include more features. In the future, we also plan to:

  • allow researchers with programming skills to write their own JavaScript code
  • offer the service of creating custom studies for researchers with specialized requirements

However, at the present moment, we’d encourage researchers to find creative ways to achieve their goals within the current limits of the grammar. We’d also appreciate it if you could email us at researcher.help@findingfive.com and tell us how our grammar falls short for your study, so that we can improve it.

Researcher Subscriptions

I already have a FindingFive membership, do I need to do anything to continue using the platform?

In most cases (if you have a Basic or Premium membership), no action is required from you. The Basic membership will automatically convert to the Lite plan, and the Premium membership will automatically convert to the Standard plan (with no fee increase). You will continue to enjoy access to the same features as before and will even receive some upgrades ;). For example, the Lite plan allows researchers to launch up to 3 concurrently active sessions instead of just one under the old membership structure.

If you have a Laboratory membership, your membership will be converted to the Pro plan. You will also receive a free 3-month extension based on the original end date of your membership. However, following this extended period, you will need to manually re-enroll in the Pro plan if you intend to keep a similar level of premium access.

How do I add a subscription?

Adding a subscription is now completely automated, even for those interested in the Pro plan. Just sign into your FindingFive account and navigate to the “Subscription” page under “Profile”. There you can see the pricing for different plans, compare plans, and select a plan for purchase. You can also add “Plus 1s” to a subscription at a significant discount.

What are Plus 1s?

Plus 1s enable you to add colleagues or students to your subscription plan, so that they can enjoy the same plan benefits as you do. To make it even better, Plus 1s’ usage quotas, including storage for files and participant entry exemptions (see below), are calculated separately so that you won’t share those quotas with your Plus 1s. To add someone as a Plus 1, they must not already have an existing subscription. It is possible to add Plus 1s when you first purchase a subscription or to an existing subscription.

Please note that if you add a Plus 1 to an existing subscription late in the subscription cycle (e.g., near the end of your billing month), the cost will not be prorated.

How do I pay for a subscription?

Once you’ve configured a plan, click “Activate” and add your debit or credit card information to pay. Your payment card information is securely processed by Square Inc. Once the subscription is active, it’ll renew automatically using the same card for each new billing cycle (monthly for monthly subscribers and annually for annual subscribers).

How does upgrading or downgrading a subscription work?

If you upgrade your plan, the upgraded subscription will be effective immediately. If you are switching to a plan with fewer features (e.g., moving from Pro to Standard), your subscription will go into effect at the beginning of the next billing cycle.

I am currently on a monthly subscription. How can I switch to an annual one? (or vice versa)

Just create a new annual subscription and activate it. The new annual subscription, if its plan is the same one as your current monthly one, will be scheduled to take effect at the moment your monthly subscription expires. The transition will be seamless. If it represents a plan upgrade, it will be effective immediately.

What happens when I cancel my paid subscription?

When you cancel your paid subscription, your benefits, including those for your Plus 1s, will stay active until the end of the current billing cycle. After that, your subscription will expire, you and your Plus 1s will all lose the plan benefits, and your payment card will not be charged again.

Can I get a refund on my subscription fee?

Unfortunately, we do not issue refunds for subscriptions under any circumstances.

I want to set up subscriptions for my entire lab or research group, can I do that under the new system?

Yes! That’s what Plus 1s are designed for! In addition, Standard and Pro plans offer new collaboration features that will make collaboration between lab members easier.

Participation Reward and Direct Payment

Where can I use the direct payment features?

The direct payment features are for sessions run on FindingFive only. Sessions launched on Mechanical Turk go through Amazon’s payment system.

How do I make a deposit?

You can make a deposit either from your profile dropdown menu or during the process of launching a new session. Our payment processing vendor is PayPal, which accepts PayPal payments, credit and debit cards. You do not need to have a PayPal account to make a deposit.

How do I pay participants?

In short, we take care of most of the work. When you are ready to launch a study, use the calculator in the new session wizard to determine how much it will cost to run your study. Make an additional deposit if your account balance won’t cover all the cost. Once your study successfully launches, we’ll make sure that your participants get paid.

Does FindingFive charge any fees?

Yes. FindingFive charges a flat fee of $0.75 per participant on the US server and €0.75 per participant on the EU server. These fees allow us to keep our payment system running, including by covering the fees that PayPal charges us for accepting deposits from researchers and transferring payments to participants.

When will my account balance be deducted for the cost of a session?

Upon launching the study, the funds will be immediately deducted from your account and held for participant payment. Participants who complete your study will automatically be approved for payment, which will be deposited in their PayPal accounts the next day.

What if fewer participants complete my study than I requested?

If a study session ends or is terminated before the target participant count is reached, the unused portion of funds (including costs for participants and FindingFive fees) will be automatically returned to your account in 1-3 hours after the session terminates. Please note that compensation will be held for in-progress participants until they complete the study or abandon it. You may receive your returned funds in multiple installments as in-progress participants gradually abandon your study over time.

Can I withdraw the leftover funds in my account balance?

Of course! If you wish to withdraw remaining funds from your FindingFive account, you may contact our researcher support team (researcher.help@findingfive.com), and we will provide a refund to your original payment method within 3-5 business days. A refund fee of $0.50 (US server) or €0.50 (EU server) will be deducted from the refund.

In the rare case where the refund request is made more than 90 days after the original deposit, we can no longer issue a refund to the original payment method. In such cases, we will make a direct PayPal transfer or mail a paper check (within the United States) to issue the refund. A refund fee of $1.00 (US server) or €1.00 (EU server) will be deducted from the refund.

Can I award bonuses to participants?

Yes. You can do that on the session management page.

Can I not pay a participant by rejecting their work?

No, rejection of participant work on the basis of quality is not permitted on FindingFive. You may decide to exclude the participant’s work from your data, but they still must be paid. Payment of participants will be automatic; you are not required to approve participant work for payment. You may terminate a research study at any time, but terminating a study does not mean that you will not have to pay participants who have submitted studies for payment.

How do I decide how much to pay participants?

FindingFive strongly encourages a minimum pay standard corresponding with the current USA federal minimum wage (July 2020: USD $7.25). You should calculate participant payments based on the expected average completion time of your study and plan to pay at least the minimum wage.

Please refer to our Terms of Service regarding how we may take actions against researchers who continually underpay participants.

How do I verify that someone participated in my study if they do not complete it?

You should make sure that you request contact information from all participants in the new session wizard, as shown in the screenshot below:

_images/contact-information-scope.png

Once you have done so, when a participant does not complete a study, their data are not recorded, but the amount of time they spend on a study will be. You can use this information as a basis for determining what actions to take should that participant contact you for help.

What do I do if I’m having problems with a participant?

If you are having problems with a participant, efforts should first be made to resolve the issue through direct contact with the participant. If you are unable to resolve the issue, FindingFive may, but is not obligated to, assist in negotiating a mutually acceptable agreement. If FindingFive does intervene, and a mutually acceptable agreement cannot be reached, FindingFive reserves the right to make the final decision, which may require you to issue additional payment to participants.

Session Fees

What is a session fee?

Starting October 1, 2021, we will charge a session fee that is based on “participant entries”, where a participant entry refers to a single completed participation event in a production session. This fee will apply to all studies, including course-credit studies, which were previously 100% free.

Sessions fees are based on “participant entries”, but what does this mean?

We are using the term “participant entries” mainly to highlight our policy that the same participant entering and completing in more than one session will be counted as multiple participant entries, instead of a single one.

Partial participation, including those who leave a study voluntarily, due to technical issues, or simply time out, does not count, so you will not be charged for participants whose data do not show up in the CSV file.

What is participant entry exemption?

For Standard or Pro subscribers, session fees will be waived for a certain number of participant entries per monthly subscription cycle (see details on our pricing page for the US server or the EU server). Those participant entries will not be counted when determining the balance due on a session fee bill. Participant entry exemptions are not available if you are on the Lite plan.

How are participant entry exemptions tallied?

Participant entry exemptions will be applied per monthly subscription cycle. For monthly paid subscriptions, the billing cycle is the monthly subscription cycle. For annually paid subscriptions, the monthly subscription cycle is marked by the monthly anniversaries of your signup date..For example, if your sign up date is Jan 15, your participant entry exemptions will reset on Feb 15.

How do monthly participant entry exemptions work in sessions that span multiple subscription cycles?

If your study session takes more than a month to finish recruiting, or it happens to span two monthly subscription cycles (even when it’s only a few days), participant entry exemptions allocated across multiple subscription cycles will apply, based on the time when a participant starts the session.

For example, let’s assume your monthly subscription cycle starts on the 15th of a month, and you have a subscription to the Standard Plan (which has 25 exemptions per cycle). A session is launched on the 10th and continues to be active till the 21th, spanning two monthly subscription cycles. During the 10th-14th, 13 participants started and completed the study; during the 15th-21th, 20 participants started and completed the study. Assuming you have not used any participant entry exemptions in these cycles, your final bill for this session is actually $0 - because the 13 participant entries are exempted under the first cycle’s quota, and the other 20 are exempted under the second cycle’s quota.

Will participant entry exemptions stay in effect even after I cancel my subscription?

The exemptions will still be in effect as long as your canceled subscription has a remaining term left (please see above for what happens when you cancel a subscription). In the case when a session remains active after your subscription expires, participant data collected after subscription expiration will be charged the session fee rate of the Lite plan.

When are session fees charged?

Session fees will be charged only after researchers obtain the participant data. Specifically, a session fee bill will be generated upon the first download of data from a session. Subsequent downloads of the data from that session will not result in additional fees. You can choose to pay the session fee bill immediately or at a later date.

How much will I be charged for session fees?

Session fees are calculated based on the subscription plan the researcher has at the time when participants join and start the session. If you have a Standard or Pro subscription, fees are only incurred once you’ve used up your participant entry exemptions for each billing month. If you are on the Lite plan, you will always pay session fees. For detailed session fee rates, see the pricing page on the US server or the EU server.

When do I need to pay my session fee bills?

Small session fee bills (less than $100 for the US server or €100 for the EU server) are due within 15 days of downloading the data. You will receive a reminder to pay your bill before the 15 days are up. If you do not pay the bill in time, it will be considered overdue, and you will be unable to launch new sessions until you pay the overdue balance.

It is possible to download data from multiple sessions (generating multiple session fee bills) and pay the session fee bills all at once. In other words, you are not necessarily required to settle one bill before generating another.

Is there a limit on how much session fee balance my account can carry?

Yes. Once your total outstanding session fee due reaches $100/€100, you must pay down some of your balance before you can launch new sessions again (each individual session fee bill must be paid in full). This is in parallel to the 15-day due date on individual bills. In other words, if the total session fee due in your account has exceeded $100 and you are on a tight budget, it is probably a good idea to first pay the bills that are already overdue or due soon.

How do FindingFive’s rates compare to other platforms?

Even with the new fee structure, FindingFive is the most affordable platform that integrates study creation, recruitment, and payment all in the same place. As far as platform fees go, researchers running 100 participants a year can save almost $300 and large research labs can save over $3,000 by using FindingFive! For details on cost comparisons with other research platforms, see this blog post.

Data Collection, Storage, and Identifiability

How secure is data collection on FindingFive?

FindingFive uses HTTPS, also known as HTTP over SSL, for all activities taking place on FindingFive’s web platform. This means that all data, including participants’ responses, are transmitted in encrypted forms over the Internet. The encryption methods we use are state-of-the-art and pose minimum risks in leaking participants’ data.

Where is participant data stored?

Participant data are stored on FindingFive’s cloud servers, located in Newark, NJ, Chicago, IL, and Atlanta, GA (for all our US researchers), or Frankfurt, Germany and Paris, France (for all our EU researchers).

How safe is the stored participant data?

Participant data are stored in password-protected databases, where the passwords provide a first layer of defense against data leak. Furthermore, these databases are set up behind a layer of Virtual Private Networks (VPN), meaning that only FindingFive’s team can access such databases. As a result, participant data on FindingFive are very safe as they are completely closed off, as if non-existent, to the Internet at large.

Is participant data stored in an encrypted form?

Optionally. For most studies, although the transmition of participant data is encrypted, once the data reach our servers, they are stored plainly on our servers. Researchers with a Premium or Laboratory membership can optionally create encrypted studies, where participant data not only are transmitted in an encrypted form, but also stay encrypted in our database. This practice, known as “encryption at rest”, provides a strong layer of protection for participant data as they will be complete gibberish to an unauthorized party. Please consult your local laws and regulations if such encryption is necessary for your research, especially if your studies collect sensitive personally identifiable data.

Is there any redundancy in the storage of participant data?

Of course! Data on FindingFive’s US servers are backed up in three different locations, under the same level of security as described above. Data on the EU server are backed up in two different locations.

To what extent does FindingFive remove identifiable personal information?

Once a study session is finished, you will be able to download participant data. Each participant will be identified by a special ID, non-reversably generated from their real FindingFive ID and your researcher FindingFive ID. As a result, each participant will have a persistent ID in the studies of the same researcher, but that ID differs between researchers, and does not exist in our database at all.

Under most circumstances, IRBs prohibit the collection of personally identifiable information in human subjects research. However, if a researcher has a unique situation that requires the collection of personally identifiable data, it is possible to collect this information via a response. In this case, researchers must use the Encrypted Study, rather than the Standard Study option. Researchers who collect personally identifiable information within their studies assume full responsibility for any losses or damages due to data security breaches. In the event of a data breach, researchers must promptly notify their IRBs (and other relevant authorities), as well as any affected parties. FindingFive reserves the right to require researchers to perform any action deemed necessary to remedy or mitigate the data breach or to prevent future data breaches.

Why can’t a researcher download participant data from an active session?

Downloading participant data from an active session is a major privacy concern. As participants start working on your experiment, their names and emails may be made available to you (see the next question). If downloading participant data from an active session were allowed, then it would be theoretically possible to match up the identities of participants to their data by checking the data every time a new participant completes the study. Linking identities to data is explicitly prohibited by most IRBs, and is never appreciated by participants.

As a result, in order to minimize the chance of participant data being linked to their identities, we only allow participant data to be downloaded after a session is complete or canceled by a researcher.

Do I get participants’ contact information?

It’s up to you and what your IRB allows. You can choose to get contact information of all participants, only those who complete your study, or no one at all. Importantly, their contact information is provided to you separately from the participant data, so that you can never link the contact information to participants’ data.

What if my home institution requires the collection of demographic information?

Some non-identifiable demographic information can be requested at the time of data download. We encourage you to negoitate with your IRB on this issue, however. In our experience, most IRBs are fine with collecting data from anonymous participants online.

IRB Concerns

Will the IRB at my institution give approval for running studies on FindingFive?

Does your IRB approve studies conducted on online platforms in general? If so, there shouldn’t be any problems obtaining approvals for FindingFive.

In fact, we adopt more research-friendly policies than possibly all alternatives out there: we don’t own your data (the public does if your study is funded by a government agency), and we protect participant identies for you through both encryption and obscurification.

If there is any specific concern, we’d be happy to work with you and your IRB. Feel free to contact us at researcher.help@findingfive.com for this purpose.

Misc

Why do I have a participant page as well?

We encourage everyone, including researchers, to participate in behavioral studies. You can be the participant of other researchers’ studies and help contribute to our high-quality participant pool!