Let’s assume your
company has decided that transfer efficiency (TE) is the key metric in determining
solder paste quality. Transfer efficiency is the ratio of the volume of the
solder paste deposit divided by the volume of the stencil aperture. While you
agree that TE is an important metric, you are a little troubled with the recent
results in a solder paste evaluation. Two out of 10 pastes are fighting for the
top spot and it looks like TE will be the deciding metric. Paste A had a TE of
99.5% and Paste B had a TE of 99%. So management wants to go with paste A. You
are troubled because paste A has a poor response-to-pause. If it is left on the
stencil for 15 minutes or more the first print must be discarded. This weakness
may result in 30 minutes or so of lost production time in a 3-shift operation.
However, the TE test results showed that the TE of paste A was statistically significantly better than paste B. You think about this situation and something doesn’t make sense — 5% and 99% are quite close.
You dust off your statistics textbook and review hypothesis testing. Then it hits you, with very large sample sizes, means that are closer and closer together can be statistically significantly different.
The data show that paste A has a mean of 99.5% and a standard deviation of 10%, whereas paste B has a mean of 99% and also a standard deviation of 10%. The sample sizes were 10,000 samples each. These large sample sizes are important in the analysis. The standard error of the mean (SEM) is used to compare means in a hypothesis test. SEM is defined as the standard deviation (s) divided by the square root of the sample size (n):
So as the sample
size increases, the SEM becomes smaller or in statistics lingo “tighter.” With
very large sample sizes, this tightness enables the ability to distinguish
statistically between means that are closer and closer together. This situation
was not a concern with sample sizes of less than 100, however with the modern
solder paste volume scanning systems of today, sample sizes greater than 1000
Figure 1 shows the
expected sampling distribution of the mean for samples with a TE of 99.5% and
99.0% and a sample size of 100, both have a standard deviation of 10%. Note
that to your eye you do not see much difference. However, with the means and
standard deviations the same and sample sizes of 10,000 the sampling
distributions of the mean are clearly different in Figure 2.
though, is that there is no difference in the results in Figure 1 and 2. The
tiny difference in the means (0.5%) may be statistically significant with a
sample size of 10,000, but is it practically significant? Would this small
difference really matter in a production environment? Almost certainly not.
So, with large
sample sizes, we need to ask ourselves if the difference is practical. For TE,
I think we can be confident that a difference of 0.5% is not practically
significant. But, what if the difference was 2% or 5%? Clearly,
experiments should be performed to determine at what level a difference is
With the case
discussed above, I would much prefer the paste that has a 99.0% TE and a good
Patty was looking forward to sleeping in. Normally she was up very early, sometimes before 5:30 am, after usually getting to bed too late, so she was looking forward to an alarm set at 7:45 am. The kids were off from school and Rob was taking them skiing, so all agreed a 7:45 am wake up time was reasonable. Since she had no early meetings, her scheduled 9 am arrival at her Ivy University office was also in the cards.
Patty was sleeping soundly when she heard her seven-year old twin sons shouting, “Mom! Dad! Come quickly.” At the same time, their two-year old beagle, Duchess, started barking.
Her heart pounding, Patty raced to the racket now being produced by this energetic trio. As she arrived she saw her sons and Duchess looking out of their back window to see a beautiful female deer eating from their bird feeder, just 30 feet away. The entire family was involved in a bird counting exercise and had noticed, several times, that the bird feeder was “wiped out” overnight. This mystery was now solved.
The entire family agreed that it was hard to be angry at the doe, as deer are such beautiful creatures.
Figure 1. A Female Deer at the Bird Feeder at Patty’s House
It was 6:15 am and it didn’t seem to make sense to go back to bed. So, Patty stayed up and was off to Ivy U in less than 30 minutes.
Patty had a rather light week as she had guest speakers for her two lectures. However, she was sitting in for one of the engineering school’s senior professors later in the day. This fellow prof had asked her to sub for him as he was called to an emergency meeting overseas. Her topic was manufacturing processes; one with which she felt very comfortable. But, she had to admit to being a bit nervous sitting in for one of Ivy U’s most famous professors.
As was her usual practice, Patty checked her email first. After going through the first 5 or 6, she saw an email with the subject header, “Ivy U Professor Wins Prestigious Queen Elizabeth Prize for Engineering.” As she opened the article, she was stunned as she saw a photo of the professor for whom she was substituting later in the day. The article went on to explain that this prize was like the “Nobel Prize” for engineering.
As she finished her emails she was relishing the thought of having a less hectic day and week ahead. Maybe she would even have time to read the Wall Street Journal during a relaxing lunch. Suddenly, her phone rang, startling her a little. She picked up the receiver to hear a familiar voice.
“Professor Coleman, this is your most faithful student Mike Madigan,” Madigan cheerfully said.
Madigan was CEO of ACME at large electronics assembly contractor. Patty worked at ACME before becoming a professor at Ivy U. Her husband, Rob, and sidekick, Pete, were also ACME employees, but were now all at Ivy U. Pete was a research assistant and Rob was just becoming a research professor. Although they all enjoyed their time at ACME, they were much happier at Ivy U. All three had a part-time consulting contract with ACME and Madigan was typically their main contact at their former employer.
“Mike! What’s up?” Patty said cheerfully.
“We are evaluating a new solder paste and I’m concerned we might make a mistake if we switch,” Mike responded.
“How so?” Patty asked.
“Well, we agreed that consistency in the transfer efficiency (TE) of the stencil printed deposits was the most important criteria,” Madigan began.
“That sounds reasonable as most of our past work has shown that a consistent TE is a strong determinant of high first-pass yields,” Patty responded.
“Right! But the difference between the pastes is only two percent. The old paste has a Cpk of 0.98 and the new paste 1.00,” Mike went on.
“I sense there is more to the story,” Patty suggested.
“Yeah. The new paste has a poorer response to pause,” Madigan said.
“Yikes!” Patty almost shouted.
Patty had shown, time and time again, that poor response-to-pause in the stencil printing process can hurt productivity and lower profitability considerably.
“My sense is the two percent difference in Cpk, might not be significant,” Mike suggested.
“Mike, I think you are on to something. What printing specs were you using and how many samples did you test?” Patty asked.
“The lower TE spec was 50% and the upper 150%. We tested 1,000 prints,” Madigan answered.
“Let me do some homework and I’ll get back to you,” Patty said.
“One problem. Can you get back by 3 pm today? The new solder paste supplier is coming for a meeting at 4PM and is pressing us,” Mike pleaded.
“OK. Will do,” Patty said, sighing a bit.
“There goes my somewhat relaxing day,” she thought.
It was a good thing she had already prepared her lecture and that it was scheduled for 4:30PM.
She did finish early enough that she could read the WSJ over lunch, marveling, as always, that she was the only person her age that enjoyed reading a real newspaper.
She called Madigan at 3 pm.
“Mike, I think I have your answer. I found a formula to calculate the confidence intervals of Cpks,” Patty started.
“And the answer is?” Madigan asked expectantly.
“The Cpk 95% confidence interval on the new paste is 0.95 to 1.05, however the old paste is 0.93 to 1.03,” Patty began.
“So, even I can sense that they aren’t different,” Mike commented.
“Yes, since the confidence intervals overlap, they are not statistically different,” Patty agreed.
Figure 2. The Confidence Interval of the Cpk on the New Paste is 0.95 to 1.05.
They chatted for a while and Madigan asked if Patty could join the first 20 minutes of the meeting by teleconference. It was a bit close to her lecture start time, but she agreed.
Patty had met Madigan’s son at West Point when she visited there to be an evaluator for a workshop two years ago. She decided to ask how he was doing.
“Mike, how is your son doing at West Point?” she asked.
“Thanks for asking. He is now a Firstie and was in the running for First Captain, but he just missed it. It’s a good thing he takes after his mom,” Madigan proudly responded.
“Wow! That’s great,” Patty replied.
“I have to admit though, my wife and I are a bit nervous. He has chosen armor as his branch and there is a good chance he will see combat sometime in his career,” Madigan responded with a bit of concern in his voice .
They chatted for a while more and Patty was touched to see so much humanity in Mike Madigan. He seemed much changed from his gruffness of earlier years.
As always, some of this story is based on true events