The ultimate guide to black box testing

As developers examine their own code, they bring their own biases to each test, frequently limiting their ability to see software from a user’s perspective or testing the application in unexpected ways. That’s where black box testing can deliver real value.

What is black box testing?

Black box testing is a category of testing techniques that examines software applications from the outside, without any knowledge of the design or structure of a software system.

Black box tests can happen at a variety of levels, from unit testing that examines the functionality of bits of code to integration testing, system testing, or acceptance testing.

In a black box test, testers examine inputs and outputs to make sure an application operates as it’s supposed to, and to discover any errors that should be fixed.

Black box testing may reveal:

• Functions of the software that aren’t working properly
• Errors that occur when accessing databases
• Issues with performance, application behavior, or the ability to scale
• Problems with the user interface
• Errors that happen as particular functions are starting or ending

An advantage of black box testing is that the tester doesn’t need technical or coding know-how.

Why people use black box testing

An advantage of black box testing is that the tester doesn’t need technical or coding know-how. Anyone can do it. Because they’re only looking at one thing (“if Y does X”), testers can find and report issues quickly. It’s the behavior testing of software.

White box testing techniques, on the other hand, are functional tests of the structure of the code. In white box testing, the tester is typically a software engineer and has knowledge of how the technology works—the internal structure or code.

You could test an algorithm, for example, with white box testing. Whereas with black box tests, you can only test user functions.

Black box testing is used because anyone can check to see if a feature is functioning as intended. If the test fails, it’s up to the engineering team to explore what went wrong.

Popular black box testing techniques

Black box testing techniques can be functional or extra-functional. Here are the typical black box testing techniques used:

1. Equivalence class partitioning

Designed to reduce the total number of test cases to be developed (therefore reducing redundancies and time to test) by defining test cases that uncover classes of errors.

It divides the input data of a software unit into partitions of equivalent data from which test cases can be derived.

2. Decision table testing

A visual matrix depicting different sets of input combinations and their expected outcome on each row. They test an application’s behavior based on specific inputs.

3. Cause-effects graphs

Graphically illustrates the relationship between effects and causes, identifying all the factors that influence the particular outcome.

4. Boundary value analysis

The most commonly used black box testing technique. A form of functional testing focused on determining and testing at chosen boundaries for input values. Boundary values include just inside/outside of the boundaries, typical values, error values, maximum, and minimum.

5. Error guessing

The software tester uses their past experience to determine where errors in the software may be. Test cases are then designed to find those errors and any lingering bugs.

Benefits and limitations of black box testing

Benefits of black box testing

1. Objectivity

In a black box test, the tester is completely separated from the developer who created the code, providing critical distance and ensuring that testers use and test the application in ways the developers had not considered.

Testing teams must view the software from the perspective of an actual user, ensuring that the software is more responsive to the needs of users.

2. User-focused

Testing teams must view the software from the perspective of an actual user, ensuring that the software is more responsive to the needs of users.

3. End-to-end testing

Because a black box test looks at all relevant aspects of a software system from a user’s perspective, tests are better able to determine the end-to-end functionality of elements such as databases, dependencies, user interface, user experience, web servers, application servers, and integrated systems.

4. No technical knowledge required

Black box testers don’t need specific technical knowledge, programming skills, or IT backgrounds. For this reason, tests can be easily outsourced or crowdsourced.

Limitations of black box testing

1. Test coverage. It’s impossible for black box testers to completely test everything in large and complex projects. In contrast, white box testing allows testing teams to focus their limited time and resources on areas that are most likely to have issues.
2. Overlapping effort. A black box approach may repeat tests that have already been performed by developers.
3. Challenges designing test cases. With limited testing time, it’s harder for testers to identify all potential inputs, making it more difficult and time-consuming to write test cases.

Black box vs. white box testing

How does black box testing compare to white box testing? In black box testing, we don’t make any assumptions about the inner workings of the system we’re testing.

Whereas in white box testing, we test the internal control flow of the target of our tests. In the case of software, this means the tester can look at the code and design tests based on the possible code paths they see.

The test can also verify that certain side effects occurred when executing the test.

By looking at the inner workings of the code, white box tests can discover specific edge cases. But a tester might also miss potential test cases that weren’t apparent by looking at the code, but that were part of the requirements.

By focusing only on the implementation, we might be “tricked” into thinking this is the correct implementation and design our tests according to the code we see, instead of the specifications that were requested.

What is gray box testing?

It should be clear now that both methodologies have advantages and disadvantages. That’s why a software development team doesn’t have to choose one or the other.

Most teams use a combination of both. This is called gray box testing. It gives testers all the necessary information to come up with the most complete test suite possible.

Types of black box testing

1. Functional black box testing

Functional testing is done when we test our system for the functionality that we implemented. This is usually at a higher level, like user interface testing, end-to-end testing, smoke testing, or user acceptance testing.

But we can also test the implemented functionality of our software during integration testing or unit testing. When we do so, we can test the system without knowing its inner workings.

We can start entirely from the requirements and specifications. We can use these to identify test cases, execute these cases, and verify the results. That’s what makes these tests black box tests.

Often, we will repeat these tests long after the features have been implemented and released. The tests are then part of our regression testing, ensuring features keep working as intended and don’t break because of new changes.

2. Extra-functional black box testing

In extra-functional testing, we test parts of our application that don’t really have a functional impact on our users. If the tests fail, the software may still be able to solve the user’s problems—but maybe it won’t work as well as it usually does.

Examples of extra-functional testing are usability testing, load testing, performance testing, scalability testing, security checks, and crash recovery testing. Extra-functional tests are mostly black box tests.

These types of tests don’t concern themselves with the specific implementation of the software, but rather check for certain issues present in the system as a whole.

3. Regression black box testing

Regression testing examines an application to see whether changes, updates, or upgrades have altered the existing functional and extra-functional capabilities of the software.

Black box techniques

1. Equivalence class partitioning

Equivalence class partitioning divides input values into different classes or groups based on the similarity of outcomes. This technique helps to improve test coverage while reducing rework and time spent.

The idea behind this technique is that within any given class, all values are interchangeable.

So, if you tested with a single value from that class, testing with more values would only add redundancy. By using equivalence class partitioning, you end up with greater test coverage while running fewer tests overall.

Let’s look at an example. Imagine you’re writing a banking application in which you need to calculate taxes on capital gains. You have the following tiers for capital gains:

• $0 – $47,025 -> 0%
• $47,026 – $518,900 -> 15%
• $518,900 -> 20%

The idea here is that by testing one value from each bracket, you’re good. So, you could verify with:

• $10,000, expecting 0%
• $100,000, expecting 15%
• $600,000, expecting 20%

Boundary value testing looks for errors in input values that may range from both ends of a boundary, where applications typically have more issues.

2. Boundary value testing

Boundary value testing looks for errors in input values that may range from both ends of a boundary, where applications typically have more issues.

When writing code, programmers often must perform comparisons. When doing those, it’s quite easy to make a mistake and replace, say, a greater-than sign (>) with a greater-than-or-equal sign(>=), or vice versa. This error is so common that it has a name: off-by-one error.

This coding error causes the application to perform a certain action one time more or less than it should.

Or it might cause the program to accept values that are outside the intended acceptable range—for instance, to accept -1 or 11 as valid when the range should be 0 to 10, both ends inclusive.

This is where boundary value testing comes in handy. This technique consists of testing:

• The value at the boundary
• The successor of the boundary value
• The predecessor of the boundary value

So, for our example where the accepted range is from 0 to 10, we would test with the following values:

• -1, expecting to see an error or validation message
• 0, expecting it to work
• 1, expecting it to work
• 10, expecting it to work
• 11, expecting to see an error or validation message

3. State transition testing

State transition testing is a technique that examines the behavior of the system under different or changing states.

There are systems that can be in different states and can transition between those states, based on things like events, inputs from the user, or the passage of time.

It’s common for such systems to have strict rules regarding what can or cannot happen when transitioning from one state to another. Unfortunately, it’s also common for bugs to hide in these transitions.

Example: CMS workflow and state transitions

For instance, suppose you work on a content management system (CMS) in which a piece of content (such as an article) can be in one of the following states: draft, being reviewed, approved, scheduled, published, and archived.

The workflow is:

• The author starts writing a piece of content (draft)
• The author finishes their piece and submits it for review (being reviewed)
• The editor either asks for changes (back to draft) or approves it (approved)
• Editor schedules the post for automatic publication later (scheduled) or publishes it manually (published)
• If necessary, the editor removes an already published post from public view (archived)

Bugs can happen during the transition of these states.

For instance, what happens if someone tries to publish a post that is currently in draft? What if someone tries to publish a post that is currently scheduled for publication? Could you archive a piece of content that hasn’t been published yet?

Of course, you should also test that only authorized roles can perform certain transition changes. For instance, the author should be able to submit their piece for review (draft -> being reviewed) but should not be able to approve it (being reviewed -> approved), let alone publish it (approved -> published).

In short, it’s vital for this technique to cover that:

• The valid state transitions work correctly
• Invalid transitions don’t happen
• Users can only perform the transitions their roles have permissions to do

When you have many input parameters, the number of all possible combinations grows exponentially.

4. All-pairs testing

The motivation behind this technique is to address a specific scenario: when you have many input parameters, the number of all possible combinations grows exponentially. This makes it impractical to test all possible combinations.

All-pairs testing (also called pairwise testing) addresses this by verifying that all possible pairs of input values are covered by the tests, instead of all possible combinations of three, four, or more values, thus reducing the number of test cases you’d need to run.

Let’s see an example. Suppose you’re testing an export function that has the following several options:

• Format: .pdf, .csv, and Excel
• Date range: last 7 days, last 30 days, last 12 months
• Filters: all data, filtered by region, filtered by user
• Timezone: UTC, local

So that would be 54 test cases. How does all-pairs testing address that?

Let’s start with a format. For instance, .pdf. We have to ensure .pdf is combined with each one of the date options, filter options, and timezones, like this:

• .pdf + last 7 days
• .pdf + last 30 days
• .pdf + last 12 months
• .pdf + all data
• .pdf + filtered by region
• .pdf + filtered by user
• .pdf + UTC
• .pdf + local

And then the same for the other formats.

How pairwise testing reduces combinations

But then comes the part that’s both crucial and a bit hard to understand: doing pairwise testing doesn’t mean we go and write one test case for each of those pairs. Instead, we strategically write test cases that cover more than one pair per test case.

For instance, the test case .pdf + last 7 days + all data + UTC successfully covers four of the pairs we need to cover. I just used this handy online tool for coming up with pairwise test cases. I entered the following:

• File format: PDF, Excel, CSV
• Date range: last 7 days, last 30 days, last 12 months
• Data filters: all data, filtered by region, filtered by user
• Timezone: UTC, local

And it gave me the following result:

As you can see, these are 10 test cases, instead of the 54 we’d need to test all possible combinations. Every possible pair of values appears at least once across the 10 test cases, which is what this technique guarantees, and that gives us confidence that we are well covered.

5. Decision table testing

Decision table testing uses different input combinations to examine the behavior of the system, capturing data in a table.

This technique is particularly useful when you have several conditions that can be in one of several possible values, and all of these conditions affect the output.

So, even though it’d be possible to describe those combinations and their outcomes in prose, that would get lengthy and confusing quite soon.

It’s better to capture those scenarios in a visual, easily digestible way, and that’s what decision table testing is all about.

For example, the table below shows some scenarios for an e-commerce shipping calculation, based on the value of the order,the customer’s membership, and the desired shipping speed:

The table could go on, but you got the gist. Each row describes, in a concise way, a specific scenario, with a combination of conditions and their outcome.

Combining different techniques

There is no reason why you can’t or shouldn’t combine one or more techniques. In fact, it’s a great way to ensure your testing is solid and has great coverage.

For instance, it makes a lot of sense to apply boundary value testing when using equivalence class partitioning.

For the values that are at the border of each tier/layer/bracket, you can apply boundary value testing, verifying not only the value at the boundary, but immediately before and after it as well.

If you want to use decision table testing, it might be valuable to first apply pairwise testing to come up with a strategy for test cases.

That way, you first come up with a reasonably sized list of test cases that are worth testing and capture those in a decision table. This ensures you don’t end up with a gigantic table due to the exponential growth in combinations.

Applying black box testing to the entire test pyramid

The test pyramid is a popular concept regarding how an organization should budget its testing. Even though he didn’t create it, Martin Fowler is the one responsible for popularizing the idea.

The test pyramid is a way of thinking about how different kinds of automated tests should be used to create a balanced portfolio. Its essential point is that you should have many more low-level UnitTests than high level BroadStackTests running through a GUI. – Martin Fowler, Test Pyramid

We will now see how instead of thinking of black box testing as a separate “type” of testing, it can be useful to think of it as an approach that you can apply to the types of automated testing that you already use.

Or, alternatively, how to classify existing tests according to the taxonomy of black, gray, and white box testing.

For that, we’ll use the test pyramid as a guide, starting at its base: unit tests.

Unit testing

People usually put unit testing in the black box testing bucket. That makes sense. You’re testing specific units to verify if their behavior is correct, and usually the tests don’t care about and don’t rely on internal implementation details.

However, you can do unit testing in a way that’s closer to the white box end of the spectrum. That’s when you write tests that make heavy usage of mocks and rely on interaction verification.

That is to say, instead of asserting the actual result against the expected value, you are verifying that a certain mocked dependency got called the way you expected, with the right parameters, and the correct number of times.

Which way is better? Both have their usages. Interaction-based tests are useful particularly in situations in which the system under test (SUT) depends on a dependency that you don’t own, and also the method you’re trying to test doesn’t return anything.

In that way, there’s really no way to do state-based testing because there’s no result you can assert against.

Those tests could be considered white box (or, at least, gray box) because they know about internal implementation details. This fact makes those kinds of tests more fragile, because they can break when those implementation details change.

Black box unit tests, which are based on state/result verification, are thus more reliable and should be preferred whenever possible.

Integration testing can also happen at various points along the white box to black box spectrum.

Integration testing

Integration testing can also happen at various points along the white box to black box spectrum.

They are usually black box—you can, for instance, test controller endpoints and assert against the result, verifying that the endpoint returns the expected HTTP status codes and the correct payload.

All of this can be done without using mocks, leveraging tools like Docker and test containers to spin up real dependencies, such as databases, third-party APIs, or messaging buses.

You can make this test white box if we add, for instance, a step in which we go to the database to verify whether the data landed correctly in the right tables. However, that’d start to blur the lines between integration and end-to-end testing.

End-to-end (E2E) testing

E2E testing is probably the type of testing that is mostly squarely placed into the black box category. After all, with this type of testing, testers usually don’t care—and often don’t even know—how the system under test has been implemented.

However, it’s also possible to perform E2E testing in ways that move it closer to white box thinking. For instance, imagine an API endpoint.

You send a request to it, which, when successful, adds a new row to a certain database table, publishes a message to a given topic on the team’s messaging bus, and sends an email.

Verifying that the email got sent is still usually very much a black box approach.

But when it comes to verifying if the database table got the new row and the messaging bus topic received the message, then this can cross the boundary to white box testing, because those two things are components of a distributed software architecture.

Being aware of their existence and how the endpoint request affects them betrays knowledge of the internal implementation, and thus, cannot be considered pure black box testing.

How black box testing is performed

Steps involved in a black box test include:

1. Determining requirements

First, you need to identify the system requirements and specifications to determine what elements can be tested. This is crucial when working with black box testing because, in a way, requirements are all you have to work with.

Use acceptance criteria from your user stories, read design documents, issue requests for comments (RFCs), and talk to relevant stakeholders, such as customers/users, the project manager, or product owner for the product.

Do whatever you can to have a solid and reliable list of requirements.

Determine what kinds of tests will reveal how well the software meets requirements and decide how success will be measured.

2. Planning tests

Determine what kinds of tests will reveal how well the software meets requirements and decide how success will be measured.

Consider using a risk-based approach when deciding which areas of the application to cover first.

Even with the help AI can give you nowadays, you probably won’t be able to test the whole system from the start, so prioritize according to risk. Focus on the areas of the application that concentrate critical user workflows and are changed more often.

3. Creating test cases

This is arguably the most important step, in which you actually create the test cases for your testing. Here you’ll use the techniques mentioned earlier, such as decision table testing, boundary value testing, equivalence class partitioning, and so on.

Use the techniques at your disposal to write test cases with the most comprehensive coverage possible. If you leverage the pairwise technique mentioned earlier, you can ensure a higher coverage with reasonable effort.

Again, use a risk-based approach, prioritizing test cases for the most critical areas of the application.

4. Executing tests

Test execution is the step that you can and certainly should automate as much as possible. Use human creativity to work on creating test cases, but leave the execution for machines.

Of course, for some areas, manual testing still makes sense, but for everything else, automate relentlessly.

If you go the automation route, don’t forget to add the tests to your CI/CD pipeline. That way, the team can be alerted when the tests fail, which prevents bugs from making it to production.

5. Reporting results

This is another step that should be automated. Ideally, your black box tests will be added to your CI/CD pipeline. Upon failures, all relevant team members are informed via email, Slack, or another communication channel.

The report should contain useful information, such as:

• Which tests failed
• The failure message
• The expected value vs. actual result

Tools used in black box testing

We mentioned that black box testing is ideal for higher-level tests like UI testing and end-to-end testing. Of course, we can perform these tests manually by using our application, but that doesn’t scale well. So, which testing tools can help us more easily create black box tests?

A tool that implements the Gherkin language (like CucumberJS or SpecFlow) makes it easy to write up tests without the need to know a programming language.

It forms an abstraction over code that automates the application. Although we can also write white box tests with the Gherkin language, it’s ideal for higher-level black box tests.

Another type of tool is a record and playback tool. Tools like Selenium, Appium, and Waldo allow testers to use an application and record the steps they went through.

Later, the tool can run through these steps again and perform any verifications the tester specified. There is no need to know anything about how the features were implemented.

Because black box testing can be used at all levels of software testing, we can also design these tests with unit testing libraries.

Black box testing and agentic AI testing

Black box testing is certainly well suited for automation. That’s always been the case, even before AI, since this approach to testing doesn’t rely on knowledge about internal implementation details, but only on inputs and observable effects.

Agentic testing takes this to the next level. With the right tools, you can do a lot.

Agentic test creation requires zero knowledge about the internals of the system; you could consider it the purest form of black box testing possible.

Agentic test creation

Honestly, this has always been the dream example of black box testing. With agentic testing tools, you can generate test cases from requirements (for instance, user stories) or prompts in plain English.

This requires zero knowledge about the internals of the system; you could consider it the purest form of black box testing possible.

Agentic test automation

Testing maintenance has long been a burden for many teams. Keeping tests up-to-date and working can be a drag on productivity, especially when using E2E tools and approaches that often create very fragile tests that break very easily after changes to the UI.

Agentic testing solves that by having automation that adapts to change.

In other words, AI-powered testing tools create tests that are self-healing; they change and adapt along with your application in such a way that they become immune to changes in IDs, names, or CSS classes.

That removes the burden of test maintenance from the shoulders of your team members.

Continuous performance validation

As we’ve seen before, black box testing covers both functional and extra-functional forms of testing. Performance testing belongs to the extra-functional bucket, and is certainly something AI helps with as well.

AI agents are capable of designing, running, and assessing the results of performance workflows. Beyond that, they can continuously monitor them, so performance issues can be flagged and handled as early as possible.

Black box testing in review

Black box testing is a technique of testing where we test our system without looking at the inner implementation details.

It differs from white box testing, where we look at the implementation to identify new test cases. It’s a form of behavioral testing because we verify the external behavior of the system. Most teams use a combination of both, i.e. gray box testing.

To discover new test cases, we can use techniques like equivalence partitioning, boundary value analysis, decision tables, state transition diagrams, and error guessing.

These techniques don’t require testers to look at the implementation, and as such, don’t require technical knowledge.

Black box testing is ideal for both functional and extra-functional testing. It can cover all the way from user acceptance testing and UI tests to performance and security testing. We can even apply the black box testing technique to lower level tests like integration and unit tests.

Black box testing with Tricentis

Tricentis products offer the ability to automate functional testing, regression testing, and extra-functional testing, as well as orchestrate these tests with scalable test management, which supports black box testing as well as manual, exploratory, and automated testing with any tool.

Providing centralized control and visibility throughout the software development lifecycle, Tricentis allows QA and development teams to approach testing more strategically and collaboratively, leading to faster and higher-quality software releases.

Tricentis offers automated software testing solutions and test management tools that support continuous integration and a wide variety of testing methodologies, including data integrity testing, app-native testing, performance testing, and chaos engineering.

As developers examine their own code, they bring their own biases to each test, frequently limiting their ability to see software from a user’s perspective or testing the application in unexpected ways. That’s where black box testing can deliver real value.