(b)

Multivariate Regression Models

I. General In practice, where several responses are of interest, it is good policy to examine the dependence of each response marginally on the covariate x. For example, in the pharmaceutical example, one would normally examine how the cure rate, the principal response, is affected by treatment and other incidental variables. Subsequently, we then examine interactions among the responses. The interactions can be very important. For example, if the interaction is such that those who are cured of the disease are largely affected by side-effects, the value of treatment would be greatly diminished.

Note:
In the pharmaceutical example, since the response target effect is polytomous with 3 ordered categories, the proportional odds model

r j (x ) log = j x , j = 1, 2, 1 r j (x )

can be used. The above modeling is informal. A more formal way for 3 responses A, B, and C is as follows. For any given value of the covariate x, we may write

ijk ( x ) for the probability that A = i , B = j , C = k ,

i = 1, K , k A ; j = 1, K , k B ; k = 1, K , k C . Then, since the primary

interest usually is focused on the marginal dependence of each response on X, we first make a linear transformation from

ijk ( x )

to new probabilities rijk ( x ) by r = L , where L is a matrix of 0 and 1 only. It is usually sensible to choose L such that

r = 1 L kC
kB kC

11 L k k
kC k =1

B C

111 L k k k

A B C

],
t

where

i = ijk , ij = ijk . Thus, in the 2 2 2 case,

j =1 k =1

k A = k B = k C = 2 , there are 6 univariate marginal probabilities,

( 1

2 1 2 1 2 ) ,

12 bivariate marginal probabilities trivariate marginal probabilities

( 11 ( 111

12 L 22 ) 112 L 222 )

and 8 . In

addition, we also need to take the interactions of the responses into account. Next, we can make the transformation

= C log(r)

= C log(1 ) L log kC

L log(111) L log kAkBkC

)]

for an appropriately chosen constant matrix C. For example, in the

2 2 2 case discussed earlier, we can take

logistic factorial contrasts,

to be the vector of

1 1 a = log( 1 ) log( 2 ) = log = log 1 1 2 = logit( 1 )

b = log it ( 1 ) c = log it( 1 )

ab = log ( 11 ) log ( 12 ) log ( 21 ) + log ( 22 )
11 21 log = log 12 22
2

ac = log(11 ) log(12 ) log( 21 ) + log( 22 )

11 21 = log log 12 2 2

bc = log( 11 ) log( 12 ) log( 21 ) + log( 22 )

11 21 = log log 12 22

abc = log(111) log(121) log( 211) + log( 221) log(112 ) + log(122 ) + log( 212 ) log( 222 )
111 211 121 221 = log log log log 212 122 222 112
Finally, we regression on the covariate x by

a (x ) = x a , b (x ) = x b , c (x ) = x c . ab ( x ) = ac ( x ) = bc ( x ) = abc ( x ) = 0 ,

For the interaction term, we can choose

to represent a linear logistic regression on x and that the 3 responses are mutually independent. Another simple model is to choose

ab ( x ) = ab ,ac (x ) = ac ,bc ( x ) = bc ,abc ( x ) = abc ,

represent a model with the interactions among the response being independent of x.

Note:
In the above model, the same set of covariate x has been included. The choice may often be reasonable but it is not necessary and there
3

may well be circumstances in which it is reasonable to exclude a particular covariate from one marginal regression model and include it in another.

Note:
If A, B, and C were each ordinal, we would replace

with the

cumulative univariate, bivariate, and trivariate probabilities. In the case of a bivariate ordinal response, it is most natural to define the model

a = log it(ri ) = logit[P( A i )] b = log it(r j ) = logit[P(B j )]

ab = log(rij ) log(ri rij ) log(r j rij ) + log(rij )

rij r j rij log = log r r r i ij ij P( A i, B j ) P( A > i, B j ) log = log P( A > i, B > j ) P( A i, B > j )
where

ri = P ( A i ), r j = P (B j ), rij = P ( A i , B j ) rij = P ( A > i , B > j )

For the interaction terms, the following choices are available,

ab = 0,ab = ,ab = i ,ab = i + j ,

or dependence on covariate is also possible. II. Log-linear regression model

A important special case of the model considered in the previous section is obtained by taking

L = I

and

C = I

. Then, the

log-linear model in which the log probabilities

ijk (x ) = log[ ijk (x )],

can be expressed as linear functions in covariates x. For example, if (A, B) is a bivariate response, the model

log ij (x ) = ( )ij + xi + x j ,

can be used.

Note:
In the bivariate binary case, we take

L = I

and

to be the

usual matrix of factorial contrasts such that

a = log(11 ) + log(12 ) log( 21 ) log( 22 ) b = log(11 ) log(12 ) + log( 21 ) log( 22 )

ab = log(11 ) log(12 ) log( 21 ) + log( 22 )

a , b

Thus,

and

ab

can be reduced to

( x ) = 0 + x a

b ( x ) = 0 + x

( x ) = ( ab

where

 0 = ( )11 + ( )12 ( )21 ( )22

0 = ( )11 ( )12 + ( )21 ( )22

( ) = ( )11 ( )12 ( )21 + ( )22

= 21 22 = 21 2 2
The above
a , b ab

and

can be derived as follows:

a = log(11 ) + log(12 ) log( 21 ) log( 22 ) = ( )11 + x1 + x1 + ( )12 + x1 + x2 ( )21 x2 x1 ( )22 x2 x2 = [( )11 + ( )12 ( )21 ( )22 ] + (21 + 22 )x

Similarly,

and

ab

can be simplified.

(x) = 0 + x ,b (x) = 0 + x ,ab (x) = ( ) a

and

log ij (x ) = ( )ij + xi + x j

are entirely equivalent in expressing the same model including two additive factors depending linearly on covariate x and interaction terms.

Note:
In previous section, for 2 2 example, the vector of logistic factorial contrast based on marginal probabilities is

 ab = log ( 11 ) log ( 12 ) log ( 21 ) + log ( 22 )

a = log ( 1 ) log ( 2 ) = logit ( 1 ) = 0 + x b = log ( 1 ) log ( 2 ) = logit ( 1 ) = 0 + x

11 21 = log log 22 12 = ( )

This model is different from the model

log ij (x ) = ( )ij + xi + x j .
logit[ 1 ( x )]
is nonlinear in x.

The key differences include z The latter case implies that

z The former case expresses the logistic factorial contrasts of the response probabilities linearly in x while the latter case can be viewed as a linear model for the factorial logarithmic contrasts. z The former case is based on the marginal probabilities while the latter case is based on the log-probabilities. z The maximum-likelihood fitted values for the models in the two cases are different in most situations.