What Is TestNG: Benefits, Annotations, and How to Use It

Introduction

Writing an automated test is usually the easy part. The real difficulty begins when the test suite grows.

You need to decide which tests should run, prepare the test environment, supply different sets of data, manage failures, run tests in parallel, and make the results easy to understand. Selenium can automate browser actions, but it does not manage these broader testing requirements on its own.

This is where TestNG comes in.

TestNG is a Java testing framework that gives developers and QA teams more control over how automated tests are organized and executed. It supports test grouping, dependencies, parameterization, parallel execution, lifecycle annotations, listeners, and configurable test suites.

This guide explains what TestNG is, the main TestNG features, its benefits, how TestNG annotations work, how to create a test suite, and why the framework is commonly paired with Selenium.

What Is TestNG?

TestNG is an open-source testing framework for Java. The name stands for Test Next Generation.

It is designed to support a wide range of testing requirements, from isolated unit tests to integration, functional, and end-to-end tests. TestNG draws inspiration from earlier Java testing frameworks while adding more flexible options for test configuration and execution.

The TestNG framework helps you:

• Identify test methods through annotations
• Run setup and cleanup methods at different stages
• Organize tests into logical groups
• Pass parameters to tests
• Run the same test with multiple data sets
• Define dependencies between tests
• Execute tests in parallel
• Configure test suites through XML
• Generate test reports
• Extend test execution through listeners

TestNG does not perform browser automation by itself. When used with Selenium, Selenium controls the browser while TestNG manages the structure, execution, assertions, configuration, and reporting around the tests.

For example, Selenium can open a login page, enter credentials, and click the sign-in button. TestNG can determine when that test runs, which browser configuration it receives, whether it belongs to the smoke or regression group, and what should happen before and after execution.

Benefits of TestNG

The advantages of TestNG become more noticeable as an automation project grows beyond a few test classes.

1. Better Control Over Test Execution

TestNG provides lifecycle annotations for running configuration methods before or after a suite, XML test, class, group, or individual test method.

This makes it easier to place setup and cleanup logic at the correct level. A database connection might be created once for an entire suite, while a browser session may be created separately for every test method.

2. Faster Feedback Through Parallel Testing

TestNG can run methods, classes, XML tests, instances, or separate suites concurrently.

Instead of executing a large regression suite one test at a time, teams can distribute tests across multiple threads and environments. TestNG provides XML attributes such as parallel and thread-count to control this behavior.

Parallel Testing execution does not automatically make a test suite safe or stable. Shared test data, static WebDriver objects, and reused browser sessions can still create conflicts. The test architecture must be designed for concurrent execution.

3. Easier Test Organization

Tests can be assigned to groups such as:

• Smoke
• Sanity
• Regression
• Checkout
• Payments
• Mobile
• Desktop
• Production-safe

You can then run or exclude selected groups without changing the test code.

This is particularly useful in CI/CD pipelines. A pull request may run only smoke tests, while a nightly pipeline runs the complete regression suite.

4. Strong Support for Data-Driven Testing

TestNG can run one test method repeatedly with different inputs.

The @DataProvider annotation can supply multiple rows of test data, while @Parameters can inject values from XML files or system properties. Data providers can also return complex objects created in Java.

This avoids creating nearly identical tests for every username, product, account type, browser, or environment.

5. Clearer Handling of Test Dependencies

Some tests genuinely depend on earlier conditions.

For example, an order-history test may require an order to exist first. TestNG supports these relationships through dependsOnMethods and dependsOnGroups.

When a hard dependency fails, TestNG skips the dependent test instead of running it under invalid conditions. It also supports soft dependencies with alwaysRun = true.

Dependencies should still be used carefully. Too many connected tests can turn a single failure into a long chain of skipped tests.

6. Useful Execution Reports

TestNG can report passed, failed, and skipped tests along with execution details and stack traces. Parameters used during test execution can also appear in its HTML reports.

When TestNG is executed through a build tool, the exact output location and report format may depend on the configured test runner.

7. Good Fit for Java Automation Projects

TestNG works with widely used Java development and build tools. Tests can be executed through IDEs, Maven, Gradle, command-line runners, and CI/CD systems.

This makes it easier to add TestNG to an existing Java project without rebuilding the surrounding development workflow.

Also Read - Guide to Run Parallel Test Cases in TestNG

Key Features of the TestNG Testing Framework

Benefits describe what teams gain from TestNG. Features describe the mechanisms that make those benefits possible.

1. Annotation-Based Configuration

TestNG annotations define test methods and configuration methods directly in Java code.

Annotations such as @BeforeMethod, @AfterClass, and @BeforeSuite provide control over when setup and cleanup logic runs.

2. XML-Based Test Suites

A testng.xml file can define:

• Test classes and packages
• Included and excluded methods
• Included and excluded groups
• Parameters
• Listeners
• Dependencies
• Parallel execution settings
• Thread counts

This lets teams change how a suite runs without editing every test class.

3. Test Grouping

The groups attribute of @Test assigns methods or classes to one or more categories.

Groups can then be included or excluded through XML, command-line options, or build-tool configuration. TestNG also supports regular expressions when selecting groups in XML.

4. Method and Group Dependencies

TestNG can execute a method only after another method or group has completed successfully.

Dependencies describe a real relationship between tests. They should not be used simply to force an arbitrary execution order.

5. Parameterization

The @Parameters annotation injects values into test or configuration methods.

Values can come from:

• testng.xml
• Java system properties
• Programmatic TestNG configuration

XML parameters can be declared at suite, test, class, or method level. More specific levels take precedence over broader levels.

6. Data Providers

A method annotated with @DataProvider supplies data to a test method.

Data providers can also run their generated tests in parallel by using @DataProvider(parallel = true).

7. Parallel Execution

TestNG supports several parallel modes:

• parallel="methods"
• parallel="tests"
• parallel="classes"
• parallel="instances"

The thread-count attribute controls how many threads TestNG allocates. Each mode changes what TestNG keeps together in the same thread, so the correct option depends on the structure of the suite.

8. Listeners

Listeners receive notifications during test execution.

They can be used to:

• Capture screenshots after failures
• Add logs to reports
• Record test start and completion times
• Send test results to external systems
• Apply custom reporting logic
• Change test results under controlled conditions

Listeners can be registered through @Listeners, XML, Java configuration, or Java’s ServiceLoader, depending on the listener type.

9. Retry Support

TestNG supports retry logic through the IRetryAnalyzer interface.

A retry analyzer can rerun a failed test when the failure may have been caused by a temporary environment or infrastructure issue. TestNG can also create a testng-failed.xml file containing the failed tests and their required dependencies.

Retries should not be used to hide consistently failing tests. A test that passes only after repeated attempts may still be flaky or expose a real product problem.

10. Assertions

TestNG includes assertion methods for validating expected results.

TestNG supports both hard and soft assertions. A hard assertion stops the current test method when it fails. A soft assertion records the failure and lets the test continue until assertAll() is called.

Also Read: Top Features for Cloud-Based Mobile App Testing

How to Use the TestNG Framework

The easiest way to start using TestNG is through a Java project managed with Maven or Gradle.

Step 1: Install Java

Current TestNG releases require Java 11 or later.

Verify your Java installation:

You will also need:

• A Java IDE such as IntelliJ IDEA or Eclipse
• Maven or Gradle
• Basic knowledge of Java
• A project with a test source directory

Step 2: Add TestNG to the Project

For a Maven project, add the TestNG dependency to pom.xml.

The example below uses TestNG 7.12.0, the current Maven Central release at the time of writing. Teams should still check and approve dependency versions through their normal upgrade process.

For Gradle:

An IDE plugin can make execution more convenient, but the project dependency is what makes TestNG available to the test code.

Step 3: Create a Test Class

Create a Java class under src/test/java.

The @Test annotation tells TestNG that calculatesOrderTotal() is a test method.

The dataProvider attribute connects the test to the orders data provider. TestNG runs the method once for every row returned by orderData().

Step 4: Run the Test

From an IDE, right-click the class or method and run it as a TestNG test.

For Maven projects that follow standard test naming conventions:

For Gradle:

TestNG can also be called directly when it and the compiled test classes are available on the classpath:

TestNG supports command-line, Eclipse, IntelliJ IDEA, and other execution methods.

Step 5: Review the Results

Check the console output and the report directory created by your runner.

When TestNG runs directly, its default report directory is generally test-output. Maven and Gradle may place reports in their own build directories.

Review:

• Passed tests
• Failed tests
• Skipped tests
• Assertion messages
• Exception stack traces
• Parameters used during execution
• Total execution time

Also Read - Top Mobile App Testing Framework

TestNG vs JUnit: Key Differences

TestNG and JUnit are both capable Java testing frameworks. Comparisons that claim JUnit cannot run tests in parallel or does not support parameterized tests are no longer accurate.

Modern JUnit supports parameterized tests, tags, configurable execution order, extensions, test suites, and opt-in parallel execution.

The more useful comparison is how each framework approaches these capabilities.

TestNG may be a better fit when a Selenium project relies heavily on XML suites, named groups, method dependencies, and configurable parallel runs.

JUnit may fit better when a development organization already uses the JUnit Platform extensively for application-level unit and integration tests.

Neither framework is automatically better for every project. The decision should reflect the project’s test architecture, build system, team experience, and reporting requirements.

Also Read - A Guide to Unit Testing Vs. Regression Testing

TestNG Annotations

TestNG annotations control when methods run and what role they play in the test lifecycle.

The scope of @BeforeTest is often misunderstood. It does not mean before every method annotated with @Test. It runs before the methods contained in the relevant <test> section of the XML suite. Use @BeforeMethod when setup must run before every test method.

1. TestNG Annotation Execution Order

A typical execution flow looks like this:

@BeforeGroups and @AfterGroups run around the first and last methods belonging to the specified groups.

2. Common @Test Attributes

The @Test annotation supports several useful attributes:

Frequently used attributes include:

• groups: Assigns the test to one or more groups
• priority: Schedules lower priority values before higher values
• dependsOnMethods: Declares method-level dependencies
• dependsOnGroups: Declares group-level dependencies
• dataProvider: Connects the test to a data provider
• enabled: Enables or disables the test
• timeOut: Sets the maximum permitted execution time
• expectedExceptions: Defines exceptions the test is expected to throw
• invocationCount: Runs the method a specified number of times
• retryAnalyzer: Connects a retry implementation to the test
• alwaysRun: Runs the test even when a soft dependency fails

TestNG schedules lower numeric priority values first. However, priority should not replace proper test independence or real dependency declarations.

What Is a Test Suite in TestNG?

A TestNG test suite is a collection of test classes, packages, groups, or methods that are configured to run together.

A suite is normally defined in a testng.xml file. The <suite> element is the root element, and one XML file represents one suite. A suite can contain one or more <test> sections, and each <test> can contain one or more TestNG classes.

The basic hierarchy is:

The <test> element in testng.xml should not be confused with a single Java method annotated with @Test.

In XML, a <test> is a logical execution block. It may represent:

• A browser
• An operating system
• An application module
• A test environment
• A device category
• A business workflow
• A group of related test classes

A suite file can also control parameters, listeners, included groups, excluded groups, thread counts, and parallel execution.

How to Create a Test Suite with TestNG

Suppose an application has two test classes:

Step 1: Create the Test Classes

Step 2: Create testng.xml

Create the file under a suitable test resource directory, such as:

Add the following configuration:

In this example:

• <suite> defines the complete suite
• parallel="tests" runs each <test> block in a separate thread
• thread-count="2" permits two concurrent threads
• The suite-level baseUrl parameter is shared
• Each <test> supplies a different browser
• Both classes run against Chrome and Firefox

Step 3: Receive the Parameters

Parameters can be injected into configuration methods:

The @Optional annotation provides a fallback value when an XML parameter is unavailable.

Step 4: Run the Suite

You can run the XML suite through an IDE that supports TestNG.

TestNG also supports direct command-line execution when the required classes and dependencies are available on the classpath:

Maven and Gradle can also run TestNG suites. The exact configuration depends on the build tool and runner version used by the project, so teams should follow the documentation for their approved build configuration.

Step 5: Check the Results

After execution, review:

• Overall suite status
• Results for each XML <test>
• Passed, failed, and skipped methods
• Browser or environment parameters
• Thread-related failures
• Setup and teardown failures
• Logs and screenshots captured through listeners

When running in parallel, each thread should receive its own WebDriver instance and isolated test data.

Why Use TestNG with Selenium?

Selenium and TestNG solve different parts of the automation problem.

Selenium performs browser actions. It can:

• Open a browser
• Navigate to a URL
• Locate elements
• Click buttons
• Enter text
• Read page content
• Switch windows or frames
• Interact with browser alerts

TestNG manages the tests built around those actions. It can:

• Run browser setup before a test
• Close sessions after execution
• Assert expected results
• Group tests
• Supply test data
• Pass browser and environment parameters
• Control parallel execution
• Define dependencies
• Generate execution results
• Notify listeners when tests fail

A simple Selenium test using TestNG may look like this:

The alwaysRun = true attribute helps ensure that cleanup still runs when an earlier test or configuration step fails.

For larger Selenium projects, TestNG provides the structure that raw WebDriver scripts lack. Tests can be divided into smoke and regression groups, supplied with browser parameters, executed concurrently, and integrated into automated delivery pipelines.

Best Practices for TestNG

1. Keep Tests Independent

Each test should prepare its own data, perform its actions, validate the result, and clean up afterward. Independent tests are easier to rerun, debug, and execute in parallel.

2. Use Dependencies Carefully

Use dependsOnMethods or dependsOnGroups only when one test genuinely requires another. Do not use dependencies simply to force execution order.

3. Limit the Use of Priorities

Too many priorities make suites difficult to maintain. Use configuration methods for setup and dependencies for real prerequisites.

4. Isolate WebDriver Sessions

Each parallel thread should have its own WebDriver instance. Avoid sharing static drivers across tests. Larger frameworks often use ThreadLocal<WebDriver> for this purpose.

5. Clean Up Resources

Close browser sessions, database connections, files, and temporary data even when tests fail. Use alwaysRun = true on cleanup methods where appropriate.

6. Keep Configuration Outside the Code

Store URLs, credentials, browser settings, device IDs, and environment details in XML files, environment variables, system properties, or secure configuration stores.

7. nUse Consistent Test Groups

Maintain a clear set of groups such as smoke, sanity, regression, and critical. Avoid using several labels for the same type of test.

8. Keep Data Providers Simple

Data providers should supply test inputs, not execute business logic. Move complex file, API, or database operations into separate utility or data-layer classes.

How to Scale TestNG Automation with HeadSpin

TestNG manages test execution, but it does not provide the browsers and devices needed to run those tests at scale.

HeadSpin lets teams run existing Selenium and TestNG automation across real devices and browsers. TestNG continues to manage groups, parameters, and parallel execution, while Selenium connects to HeadSpin through RemoteWebDriver.

A remote driver can be created using an endpoint stored securely outside the source code:

Authentication details should be stored in a secure CI/CD secret manager rather than hardcoded.

Test Across Real Devices and Browsers

Existing Selenium tests can run across different browser versions, operating systems, screen sizes, and real mobile devices. This helps teams identify device-specific rendering, interaction, and performance issues.

Run Tests in Parallel

Separate TestNG <test> blocks can represent configurations such as Chrome on Windows, Safari on macOS, Chrome on Android, or Safari on iOS.

Each TestNG thread can create an independent remote session. Available concurrency depends on the device and session capacity assigned to the team.

Reduce Local Maintenance

Teams do not need to install and maintain every supported browser and device locally. Existing Java, Selenium, and TestNG code can remain largely unchanged apart from remote driver configuration.

Add Performance Context

Alongside functional results, teams can review available session artifacts and performance information to investigate whether a failure is linked to the application, network activity, device resources, or rendering behaviour.

Improve Failure Analysis

TestNG listeners can capture test names, parameters, stack traces, and screenshots. These can be reviewed with HeadSpin session logs and artifacts to reduce manual reproduction work.

Integrate with CI/CD

TestNG suites can run through Maven, Gradle, and existing CI/CD pipelines. Teams can execute smoke or regression tests, create remote sessions, collect reports, and fail builds when critical tests do not pass.

This allows teams to expand device and browser coverage without replacing their existing TestNG automation structure.

Conclusion

TestNG gives Java testing projects a practical way to organize and control automated test execution.

Its lifecycle annotations, test groups, data providers, dependencies, XML suites, listeners, and parallel execution options make it useful for everything from small integration tests to large Selenium regression suites.

The framework works best when its flexibility is used with discipline. Tests should remain independent where possible, WebDriver instances should not be shared across parallel threads, and retries should not be used to disguise instability.

When the suite outgrows local infrastructure, TestNG automation can be connected to HeadSpin through Selenium Remote WebDriver. This lets teams retain their existing scripts while expanding execution across real devices, browsers, and user environments.

FAQs

1. What is TestNG framework used for?

Ans: The TestNG framework is used to create, organize, execute, and report Java tests. It supports unit, integration, functional, and end-to-end testing and is commonly used to manage Selenium automation suites.

2. Is TestNG only used with Selenium?

Ans: No. TestNG is an independent Java testing framework. It can test Java classes, APIs, services, databases, integrations, and complete systems. Selenium is only one of the tools that can be used inside a TestNG test.

3. What is the difference between Selenium and TestNG?

Ans: Selenium automates browsers. TestNG manages the tests around that browser automation.

Selenium handles actions such as navigation, clicking, and entering text. TestNG handles annotations, assertions, setup, cleanup, data providers, groups, dependencies, parallel execution, and reports.

4. What are the main TestNG features?

Ans: The main TestNG features include annotations, XML suites, data providers, parameterization, test grouping, method and group dependencies, parallel execution, listeners, retry analyzers, assertions, and configurable reporting.

5. Does TestNG support parallel execution?

Ans: Yes. TestNG can run test methods, classes, XML <test> blocks, instances, and separate suites in parallel. Parallel settings can be configured through testng.xml or runtime options.

6. What is testng.xml?

Ans: testng.xml is a configuration file used to define a TestNG suite. It can specify test classes, packages, methods, groups, parameters, listeners, dependencies, parallel modes, and thread counts.

7. How can I run only a specific TestNG group?

Ans: Assign the group through the groups attribute:

Then include it in testng.xml:

TestNG also supports the -groups command-line option and build-tool-specific group configuration.

8. Can JUnit run tests in parallel?

Ans: Yes. Modern JUnit supports parallel execution as an opt-in feature through JUnit Platform configuration. The difference is that TestNG commonly exposes parallel controls directly through its XML suite structure.

9. What is the difference between @BeforeTest and @BeforeMethod?

Ans: @BeforeTest runs before the methods contained in a particular <test> section of testng.xml.

@BeforeMethod runs before every individual method annotated with @Test.

10. Is TestNG suitable for large Selenium test suites?

Ans: Yes. Its grouping, parameterization, listeners, XML configuration, and parallel execution features make it suitable for large Selenium projects. The suite must still use isolated test data, thread-safe driver management, and reliable cleanup logic.

‍