Logitech Usability Evaluation
Logitech was interested in understanding the experience of college students using their external iPad keyboard accessory, specifically: 1) usability of the keyboard with a condensed key layout and 2) viability of the keyboard and iPad as a “laptop replacement.”
My team of 4 researchers first performed an Expert Review, individually then combined. We recruited college students who participated in an initial Usability Test as new users and a second Usability Test as more experienced users. The students also responded to Diary Study questionnaires in email.
Usability Evaluation
Testing & Assessment Programs
Bentley University
2014
Product Overview
- Folio-style keyboard solution for various iPad models
- Protective cover for tablet
- Provides a standing position for the iPad in landscape orientation
- Provides tactile feedback when typing (benefit over iOS virtual keyboard)
- Connects to iPad via Bluetooth (pairing)
- Charged via USB cable
- Competitive products include iOS virtual keyboard, other external keyboards, and laptops
Logitech modified keypad
Standard Mac keypad
Methodology
Expert Review
Individual reviews / Group review
Gather insights and expert opinions on the product.
- Four experts independently reviewed the product.
- Experts discussed and combined their findings into a Group Expert Review.
- Findings were used to inform tasks for the Usability Tests and questions for the Longitudinal Study.
Usability Test 1
Interview / Tasks
Gather data on usability of the keyboard from first-time users.
- A background interview was conducted, covering current habits and preferences in school-related tasks, relevant experience and expertise, and experience with any competing input products.
- Each participant was given a Logitech keyboard to use in the study.
- Participants, as first-time users, performed tasks including device setup and common typing tasks, forming an initial impression of the product.
Diary Study
3 Emails – 2 Weeks
Gather qualitative insights on student use of the product in their daily lives and over time.
- A diary study was conducted to study students using the product in their daily lives, travel, and school activities.
- 3 email questionnaires were sent to each participant. Response formats included lists, scales, and open-ended questions.
- Participants were asked to optionally provide screenshots of their typing work and photographs of their environmental contexts.
Usability Test 2
Tasks / Questionnaire / Interview
Gather data on usability of the keyboard from experienced users. Compare the keyboard’s usability to a competing input solution.
- Participants, with two weeks of experience, were asked to perform common typing tasks and student-specific tasks.
- Identical tasks were performed using the Logitech keyboard and the iOS virtual keyboard, the primary competing input solution.
- Participants were asked to fill out questionnaires regarding the tasks and to provide ratings of the input solutions.
- A final interview was conducted to better understand the experiences in the usability session and in the study as a whole.
Recruiting and Participants
Target Demographic
- College student
- Owns an iPad of 2nd generation or later
- Performs some student core functions on the iPad
Screening
Online screener questionnaire:
- Basic demographics
- iPad model
- Tablet usage expertise level
Incentives
- Keep Logitech keyboard (about $100 value)
- $100 Amazon gift card
Selected Participants
- 8 participants
- Undergraduate and graduate students
- 2 Austin, 4 Boston, and 2 Mexico
- Mix of iPad models
- Mix of iPad expertise levels
Usability Tests: Protocol
Tasks were designed to cover general keyboard use and some student-specific tasks. Participants performed the tasks in both usability test sessions. Experts performed the tasks as part of their reviews.
Unboxing the product
Attachment and Bluetooth setup
Typing a grocery list
Composing and sending an email
Browsing to common websites
Browsing to a complex URL
Interacting with web pages
Filling out a form with a complex password
Typing a long text passage with wide character coverage
Taking notes on a short lecture video
(Moderator: Refer to Recruiting Questionnaire for general background on participant.)
How do you TAKE NOTES in class?
____ Paper
____ Laptop
____ Tablet (iPad)
____ Don’t take notes
____ Other: ______________________
What SOFTWARE do you use to TAKE NOTES?
____ Microsoft Word
____ Notes (PC)
____ TextEdit (Mac)
____ Google Drive Doc
____ Evernote
____ Apple Notes
____ Pages
____ Other: ______________________
How do you write your ASSIGNMENTS?
____ Paper
____ Desktop computer
____ Laptop
____ Tablet
____ Other: ______________________
What SOFTWARE do you use to write your ASSIGNMENTS?
____ Microsoft Word
____ Google Drive Doc
____ Pages
____ Voice software
____ Other: ______________________
How do you manage your school CALENDAR?
____ Paper calendar
____ Microsoft Outlook calendar
____ Google calendar
____ Apple calendar
____ Other: ______________________
What SCHOOL-RELATED TASKS do you do on your iPad?
____ Taking class notes
____ Reading assignments
____ Writing school assignments
____ Calendar
____ Messaging with students and teachers
____ Email students and teachers
____ Research
____ Other: ______________________
Do you take your iPad to school with you?
____ Y ____ N
How do your store or carry it?___________________
Have you ever used a typing input product other than the built-in iOS keyboard?
____ Y What? ___________________
____ N
(Moderator: Start with the keyboard product in an unopened box. Let the participant open the box during the tasks. Do not mention the word “Bluetooth” to the participant.)
Task scenario 1: “You just bought this new case and keyboard product for your iPad. I’d like you to connect it to your iPad and get the keyboard up and running. Do you have any questions?”
[Answer any questions.]
Success criteria:
The case is attached properly.
The Bluetooth keyboard is paired with the iPad.
The user types successfully in an app as a test.
(Moderator: Stop the setup task after 5-10 minutes. Teach the participant to set up the device so they can do it on their own after they leave the lab.)
Task scenario 2: “You just remembered you need to go to the grocery store and buy 4 items: eggs, bread, ketchup, and plastic wrap. Write a note on your iPad so you won’t forget the grocery list.”
Task scenario 3: “You need to tell your friend Rachel you will be 1 hour late for dinner tonight. You also would like to suggest another restaurant: The Golden Lion on 405 E. 3rd Street. Send your friend an email at the following email address: InputDevices750@gmail.com”
Task scenario 4: “Go to Amazon.com and find the greatest hits album by Wang Chung.”
Task scenario 5: “Type the following paragraph in your note taking application:”
The following sentences are “pangrams” that use every letter of the alphabet: 1) The quick brown fox jumps over the lazy dog. 2) Pack my box with five dozen liquor jugs. 3) Sixty zippers were quickly picked from the woven jute bag. 4) Crazy Fredrick bought many very exquisite opal jewels.
Instructions: “You’ve had your keyboard for a couple weeks. You should have some experience using it. I’m going to ask you to perform some common tasks on your tablet. You’ll do the tasks once using the on-screen keyboard. And, you’ll do the same tasks again using the Logitech keyboard.”
Tasks using the ON-SCREEN KEYBOARD
Task scenario 1: “You just moved to a new apartment (708 Mockingbird Lane, #201, Austin, TX 78701) and also got a new cell phone (512-963-5319). Email your new contact information to your friend Stephanie at the following email address: InputDevices750@gmail.com”
Task scenario 2a: “You are web browsing for sports news. Go to the ESPN website.”
Task scenario 2b: “You need to browse to a specific web link your teacher gave out in class. Browse to:
http://booksite.elsevier.com/9780123739339/”
Task scenario 3: “Copy and paste the introduction text from the last web page to your note taking application.”
Task scenario 4: “Sign up for an account at Box.com.”
Go to: https://app.box.com/signup/personal/
Use this password: a9b5$C12
(Don't click Submit.)
Task scenario 5: “I will play a 5-minute video of a lecture. Please take notes as if you expected to be tested on the material.”
(Moderator: Use Lecture A or B. Stop at 5 minutes. Let the participant know thinking aloud is not required while taking notes.)
LECTURE A: Khan Academy: Electoral College https://www.khanacademy.org/humanities/american-civics-subject/american-civics/v/electoral-college
LECTURE B: TED Talks: Angela Lee Duckworth: The Key to Success Grit http://new.ted.com/talks/angela_lee_duckworth_the_key_to_success_grit
Tasks using the LOGITECH KEYBOARD
Task scenario 6: “You just moved to a new apartment (5101 Lakeshore Road, Apt. Z, Austin, TX 78755) and also got a new cell phone (512-367-0926). Email your new contact information to your friend Chauncey at the following email address: InputDevices750@gmail.com”
Task scenario 7a: “You are going to shop online. Go to the Amazon website.”
Task scenario 7b: “You need to browse to a specific web link your teacher gave out in class. Browse to:
http://shop.oreilly.com/product/9781783281497.do”
Task scenario 8: “Copy and paste the introduction text from the last web page to your note taking application.”
Task scenario 9: “Sign up for an account at Box.com.”
Go to: https://app.box.com/signup/personal/
Use this password: b8c4$D21
Task scenario 10: “I will play a 5-minute video of a lecture. Please take notes as if you expected to be tested on the material.”
(Moderator: Use Lecture A or B. Stop at 5 minutes. Let the participant know thinking aloud is not required while taking notes.)
How EASY was the ON-SCREEN KEYBOARD to use?
[ Difficult 1 2 3 4 5 6 7 8 9 10 Easy ]
Comments:
What did you LIKE about using the ON-SCREEN KEYBOARD?
1. ____________________________________________________________
2. ____________________________________________________________
3. ____________________________________________________________
What did you DISLIKE about using the ON-SCREEN KEYBOARD?
1. ____________________________________________________________
2. ____________________________________________________________
3. ____________________________________________________________
How EASY was the LOGITECH KEYBOARD to use?
[ Difficult 1 2 3 4 5 6 7 8 9 10 Easy ]
Comments:
What did you LIKE about using the LOGITECH KEYBOARD?
1. ____________________________________________________________
2. ____________________________________________________________
3. ____________________________________________________________
What did you DISLIKE about using the LOGITECH KEYBOARD?
1. ____________________________________________________________
2. ____________________________________________________________
3. ____________________________________________________________
Did you PREFER the ON-SCREEN keyboard or the LOGITECH keyboard?
____ On-screen keyboard ____ Logitech keyboard
Comments:
What were your OVERALL IMPRESSIONS from using the Logitech product?
Given a choice after this study, how likely are you to use the Logitech product?
[ Very unlikely / Unlikely / Don’t know / Likely / Very likely ]
Why?
Discuss any tasks that were particularly difficult.
For note taking, rank the following in your order of preference: (1-4 where 1 is most preferred)
____ Laptop
____ Paper
____ iPad
____ iPad with Logitech case/keyboard
Why?
Anything else?
Usability Tests: Results
Top 10 Positive Findings
| Headline | Description | Category | E's | P's |
|---|---|---|---|---|
| Minimizes need for laptop | Logitech keyboard folio minimizes the need to carry a heavy laptop around campus to classes. | Industrial design | 3/8 | |
| Superior to iOS keyboard | Typing on the Logitech keyboard folio is superior to typing on the iOS virtual keyboard. | Functionality | Yes | 8/8 |
| Easy to pack and carry | iPad with Logitech keyboard folio is easy to pack and carry. | Industrial design | 4/8 | |
| Easy to connect and start | Logitech keyboard folio is easy to connect, with Bluetooth, and start using with the iPad. | Bluetooth | Yes | 2/8 |
| Magnet | The magnet line on the keyboard makes it easy to attach to the iPad. | Industrial design | Yes | 4/8 |
| Good looks | Style, color, looks, and sleek form and factor are very appealing. | Industrial design | Yes | 8/8 |
| Stand position | Stand position is convenient and appreciated. | Industrial design | Yes | 5/8 |
| Secure and safe | The iPad feels safe and secure inside the keyboard folio when using, carrying, or storing. | Industrial design | Yes | 7/8 |
| Tactile feedback | Tactile feedback of the keys is appreciated, especially when compared to the iOS virtual keyboard. | Industrial design | Yes | 6/8 |
| Availability of all keys | The availability of all keys in one layout makes the keyboard easy and efficient to use, especially when compared to the iOS virtual keyboard’s keypad switching design. | Keyboard layout | Yes | 5/8 |
Top 10 Usability Problems
| Severity | Headline | Description | Recommendation | Category | E's | P's |
|---|---|---|---|---|---|---|
| High | Small size of keyboard causes more typing errors | Users experienced a substantial increase in typing errors due to the smaller-than-standard size of the keyboard. | Reconsider the layout variations from a standard keyboard. Continue testing and improving the keyboard layout. (On the iPad Mini, consider making the keyboard slightly wider than the tablet to better match an average hand size.) | Industrial design | Yes | 3/8 |
| High | TAB as key combination | TAB as a key combination (FN+Q) is inefficient and frustrating. TAB is a commonly used key. | Separate CAPS LOCK and Q keys. | Keyboard layout | Yes | 2/8 |
| High | CAPS LOCK as key combination | CAPS LOCK as a key combination (FN+A) is inefficient and frustrating. CAPS LOCK is a commonly used function. | Separate CAPS LOCK and A keys. | Keyboard layout | Yes | 4/8 |
| High | SHIFT key size and positioning | Users were frustrated by and made many errors using the SHIFT keys. The left SHIFT key is placed in a difficult-to-reach position under the A key, as opposed to left of the A key. This positioning does not allow for natural outward rotation of the pinky. The left and right SHIFT keys are positioned asymmetrically. Depending on model, one of both SHIFT keys are small in size, making them harder to hit. SHIFT is a very commonly used key. | Move the left SHIFT key more to the left of the A key. Position the SHIFT keys more symmetrically. Increase the width of SHIFT keys as much as possible. | Keyboard layout | Yes | 3/8 |
| High | CAPS LOCK key lacks status indication | Participants found the lack of feedback for CAPS LOCK on/off status inefficient and confusing when typing. | Add a light or some other indicator of the state of the CAPS LOCK key. | Industrial design | Yes | 2/8 |
| High | Number keys are too small (short) | The small size of the number keys in the top row causes more typing errors. (On the iPad Mini, the short number key row is in close proximity to the screen. One participant often accidentally touched the screen causing unwanted screen interactions.) | Increase the height of the number key row. (On the iPad Mini, also increase the distance of this key row from the screen.) | Industrial design | Yes | 3/8 |
| High | HOME key placement | Users often press the HOME key by accident, usually when intending to press the “1” or “!” characters. | Options include: moving the HOME button to a less frequently used area; or making the HOME function a key combination so it is harder to accidentally invoke. | Industrial design | 3/8 | |
| High | Bluetooth indicator light is confusing | The Bluetooth indicator light flashes until the keyboard is connected and then it turns off. Participants expected the light to stay on while Bluetooth was connected. When the light turned off, participants thought the connection failed. | Keep Bluetooth indicator light on when connected, if possible within power consumption constraints. | Industrial design | 3/8 | |
| High | Tab functionality is not supported in apps | Tab functionality is not well supported within apps. For example, tabbing through web page forms in Safari often does not work or only partially works. | Work with application developers to support tab functionality in apps. | App integration | Yes | 2/8 |
| High | Scrolling with the cursor keys is not supported in apps | Participants expected to scroll applications and web pages using the up and down arrow keys to scroll. This functionality is often not supported. Being able to scroll without moving hands off the keyboard would increase efficiency and increase the keyboard’s ability to stand alone as an input solution. | Work with application developers to support scrolling via the arrow keys in apps. | App integration | Yes | 2/8 |
Good Ideas
| Headline | Description | Category | E's |
|---|---|---|---|
| Provide full support for Microsoft Office 365 | Microsoft Office for iPad was released in April 2014, with 12 million downloads the first week. Provide full support for Microsoft Office 365 in next iteration of the keyboard. Microsoft Office is a key reason students use a laptop instead of their iPad. Providing full support for Microsoft Office will help differentiate Logitech’s keyboard in the market. | App integration | Yes |
| Provide a mobile app that can be invoked from the keyboard | Provide an accompanying mobile app for the keyboard that can be invoked via a key combination from the keyboard. Provide detailed instructions about key functions. Allow firmware updates from the keyboard. This app could be another market differentiator for Logitech. It would provide the ability to fix or add features to the keyboard via a software update and an accompanying app. | App integration | Yes |
| Make screen brightness adjustable from the keyboard | Screen brightness is easily adjustable using the iOS Control Center. Make screen brightness adjustable from the keyboard, similar to volume. | Key function | Yes |
| Mention Bluetooth on the front of the box | The front of the box does not mention this is a bluetooth keyboard. Bluetooth is shown only as a small icon on the back of the box. | Packaging | Yes |
| Add a place to store the USB charging cable | The USB charging cable is a separate piece that could get lost easily. Find a way to store or attach it to the folio. Possibly use extra space in the crease where the case folds. | Industrial design | Yes |
| Provide tips to first time user | People are unlikely to look up documentation. Providing a popup window (or some other form) listing tips would be a good idea. Maybe there could be an option for "don't show this again." | New feature | Yes |
Setup instructions too minimal
“As far as setup procedure goes, I know pictures are really powerful but if there were a few words to describe…”
USB cable is for charging
“Is this for a mouse?”
Initial impression
“I really appreciate that it is lightweight. Oh my goodness.”
Initial impression
“I like the fact that it is sleek and light, but when it comes to using it - it feels small”
Keypad layout
“Extensive notes are a bit difficult to take. I continue to struggle with certain keys such as tab, caps lock, and anything using the shift key; in general, any of the keys on the far left hand side.”
Final impression
“Because of the Logitech keyboard, I’d say that my usage of the iPad has gone up 90%”
Logitech Keyboard
Likes
Faster
Keep eyes on lecture more
Screen is not covered
Tactile feedback
More standard keyboard layout
Dislikes
(None)
Conclusion
The Logitech keyboard is clearly superior to the iOS virtual keyboard, because its keyboard layout is closer to standard, keypad switching is not required, and its keys have a tactile quality.
iOS Virtual Keyboard
Likes
Convenient, it’s there
Special characters easy to access
Dislikes
Alters your typing style
Switching keypads
No tactile feedback
Covers screen
Freezes often
Eyes on screen, instead of lecture
Diary Study: Protocol
The diary study consisted of 3 email questionnaires over about 2 weeks.
Please list:
1. Places you used your iPad's keyboard:
2. Activities you completed using the keyboard:
3. Activities where you used your iPad’s on-screen keyboard (if any), but not your external keyboard:
4. Why did you not use the external keyboard in situations you listed in #3 above?
5. Positive experiences with the external keyboard in your daily use of your iPad:
6. Where the keyboard interfered with your daily use of your iPad:
7. Three adjectives you would use to describe the keyboard:
1. Please list:
a. Activities you completed using the keyboard:
b. Activities where you used your iPad’s on-screen keyboard (if any), but not your external keyboard (and why):
c. Positive experiences with the external keyboard in your daily use of your iPad:
d. Where the keyboard interfered with your daily use of your iPad:
Please rate the following statements.
2. It has been practical to use the keyboard and transport it throughout my day and to my classes. You...
[ ] Strongly disagree
[ ] Disagree
[ ] Neutral
[ ] Agree
[ ] Strongly agree
Because:
3. Storing the keyboard has been...
[ ] Very impractical
[ ] Impractical
[ ] Neutral
[ ] Practical
[ ] Very practical
Because:
4. Typing with the external keyboard is faster than with the on-screen keyboard. You...
[ ] Strongly disagree
[ ] Disagree
[ ] Neutral
[ ] Agree
[ ] Strongly agree
5. If you use the touch screen, the external keyboard is...
[ ] Obtrusive
[ ] Slightly obtrusive
[ ] Neutral
[ ] Slightly more comfortable than using the iPad alone
[ ] More comfortable than using the iPad alone
Because:
6. If you use the iPad with phones, charger, adapter, or any other cable plugged in, the external keyboard is...
[ ] Obtrusive
[ ] Slightly obtrusive
[ ] Neutral
[ ] Slightly more comfortable than using the iPad alone
[ ] More comfortable than using the iPad alone
Because:
7. If you press the physical buttons of the iPad, the external keyboard is...
[ ] Obtrusive
[ ] Slightly obtrusive
[ ] Neutral
[ ] Slightly more comfortable than using the iPad alone
[ ] More comfortable than using the iPad alone
Because:
8. If you use the iPad's cameras, the external keyboard is...
[ ] Obtrusive
[ ] Slightly obtrusive
[ ] Neutral
[ ] Slightly more comfortable than using the iPad alone
[ ] More comfortable than using the iPad alone
Because:
1. What did you do with the keyboard during the last few days?
Please rate the following statements.
2. It has been practical to use the keyboard and transport it throughout my day and to my classes. You...
[ ] Strongly disagree
[ ] Disagree
[ ] Are neutral about it
[ ] Agree
[ ] Strongly agree
Because:
3. Storing the keyboard has been...
[ ] Very impractical
[ ] Impractical
[ ] Neutral
[ ] Practical
[ ] Very practical
Because:
4. Typing with the external keyboard is faster than with the on-screen keyboard. You...
[ ] Strongly disagree
[ ] Disagree
[ ] Are neutral about it
[ ] Agree
[ ] Strongly agree
Because:
5. I use frequently the FN keys.
[ ] Strongly disagree
[ ] Disagree
[ ] Are neutral about it
[ ] Agree
[ ] Strongly agree
Because:
6. You have now used the keyboard for almost two weeks. Mention three adjectives you would use to describe the keyboard and why.
7. How many times did you have to use the charger during these 10 days?
8. Please add any comments you think are relevant that we haven't asked you about yet.
Diary Study: Results
Places
Home
Class
Coffee shops
Museum cafeteria
Field work (community center)
In bed
At a conference
At a seminar
Activities
Email
Write short notes
Take notes in class
Take notes from a textbook
School-related research
Write an outline
School assignments
Submit assignments
Download syllabus
Download course materials
Download files for an assignment
Planning projects
Take notes for work as an interpreter
Video calls
Web browsing
Watch tutorial videos on YouTube (for statistics)
Play music (volume adjustment, play/pause)
Watch movies (volume adjustment)
Message or text
Take photos during field work
Read textbooks
Social media
Calendar
View maps to find work locations
Record research groups as facilitator
Sign into accounts to confirm events
Search for flights
Search for new apps
Activities not used
Enter special characters
Facebook messaging
Social media (Instagram is portrait only)
Read documents/textbooks for school
Listen to music
Slide to unlock
Press Home button
Spell correction
Play games
Shopping
Apple Messages
Web browsing
It is important to be aware of activities where the keyboard is not used, in order to identify gaps where additional functionality or application integration could be provided.
The following adjectives were used by participants in the diary study emails to describe the product.
2nd-4th Generation iPad
iPad Air
iPad Mini
Participants used the following apps in usability sessions or mentioned them in diary study entries. It is important to see what apps the keyboard should integrate well with and what apps currently drive students to their laptops.
Summary of Findings
| Headline | Description |
|---|---|
| Usability Observations |
|
| Portability |
|
| “Laptop replacement” barriers |
|
| Integration with apps |
Lack of software integration by apps and iOS limit the convenience of the keyboard. Some missing functionality:
|
| International languages |
|
| Industrial design |
|
| Competitive analysis | Continued innovation and optimization are recommended. Another keyboard brand could solve the issues found in this report and possibly leap frog the product. |
Next Steps
“Laptop replacement”
We recommended focusing on problems preventing the product from becoming a “laptop replacement” when combined with an iPad.
Primary keypad issues:
- The commonly-used “left side” keys, including TAB, CAPS LOCK, and SHIFT, must be made separate keys and moved to more efficient positions to meet users’ typing efficiency needs. Expanding the key area closer to the edge of the keypad might be required, or possibly a slight increase in overall keypad width.
- The iPad Mini keyboard’s width should be increased slightly to a more usable width.
Secondary keypad issues:
- The number key row should be made taller, possibly by a small movement of the tablet base/magnet position.
- The DELETE key should be made wider. Modifications to the “left side” keys might increase available key area width.
- CAPS LOCK should be given a status indicator light.
Application support:
- Work to better support the applications students are using on their laptops, such as Microsoft Office, Blackboard, and R.